qa/tasks/vstart_runner: add LocalCluster.run

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephfs/test_nfs: fiddle with sudo

- no sudo for 'ceph' commands
- explicit sudo for _sys_cmd (things like 'rados' don't need sudo!)

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs/export: some cleanup, minor refactoring

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs/cluster: remove unused @cluster_setter

Signed-off-by: Sage Weil <sage@newdream.net>

nfs/mgr: fix help message case

Signed-off-by: Sage Weil <sage@newdream.net>

doc/cephfs/fs-nfs-export: add note about export update behavior

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: move user create/delete into helper

- Do user create or delete via a helper
- Defer until after we have validated the Export (on create or update)
- Support updates to user_id, which is needed to keep the naming consistent
and to also support changing the bucket, since the user_id is derived
from that.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: refactor _delete_user helper

Single caller, trivial functions, and caller is also short.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: refactor create_export_from_dict() helper

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: keep 'nfs export get' around for backward-compat

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: rename method

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephfs/test_nfs: test new export via apply

Signed-off-by: Sage Weil <sage@newdream.net>

doc/cephfs/fs-nfs-export: be consistent with cluster_id and _ vs -

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: addr -> client_addr for 'nfs export create ...'

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: fix tests

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: 'nfs export get' -> 'nfs export info'

This is a breaking change, but aligns the CLI verbs with the rest of ceph.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: binding -> pseudo_path

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: more revisions based on review

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: adjust NFSExceptoin errno arg

Signed-off-by: Sage Weil <sage@newdream.net>

doc/cephfs: update 'nfs export {get,apply}' docs

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: merge FSExport back into ExportMgr

This class captures no data, and the current methods cover both cephfs
and rgw exports.  There is little value to making the class separate from
ExportMgr.

Signed-off-by: Sage Weil <sage@newdream.net>

doc/radosgw/nfs: document mgr/nfs way to add/remove rgw exports

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: merge 'nfs export {update,import}' -> 'nfs export apply'

The only thing we lose is a strict 'update' that errors out if the
export didn't already exist, and we don't have any known need for that.

Fixes: https://tracker.ceph.com/issues/51265
Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: test export creation and list

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: test export_update (+ fixes)

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: test Export.validate(); several fixes

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: test that export <-> block+dict conversions go both ways

Fixes: https://tracker.ceph.com/issues/50449
Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: clean up test a bit

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs/export: fix export validation

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs/export: fix tests

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: handle option addr/client block in create_export()

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: allow multiple addrs for new exports

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: fix/finish rgw export

- create the rgw keys for the export
- cleanup the rgw user on export deletion

Fixes: https://tracker.ceph.com/issues/47172
Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs/module: clusterid -> cluster_id

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs/export: fix export_update_1 to type check

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs/cluster: fix type error

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs/export: wrap long lines

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: ExportMgr._delete_export only works for cephfs for now

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/nfs: Remove pool_ns from NFSCluster

Cause this is a singleton instance. Better to explictly pass it along.

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/nfs: Remove ExportMgr.rados_namespace

Cause this is a singleton instance. Better to explictly pass it along.

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/nfs: flake8

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/nfs: Add type checking

Also introduce RawBlock class

Fixes: https://tracker.ceph.com/issues/50816
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/nfs: Add __eq__ method to Export

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/nfs: Add some compatibility to mgr/dashboard

provide default values for `squash` and `security_label`

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/nfs: Fix whitespace handling

keep whitespace in lines starting with %url

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/nfs: Copy unit tests from mgr/dashboard

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/nfs: partially implement rgw export support

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: abstract FSAL; add RGWFSAL

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: refactor to merge 'update' and 'import' code

- update must be an existing export
- import can be a new or existing export, and we will either update or
  create as appropriate.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: add 'nfs export import' command

Allow import of JSON to create a new export.  Similar to 'update', which
allows an import of JSON to update an existing export.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: refactor 'nfs export update' and export validation

Move validation to the Export class, so it can check any object in
place.

Refactor update code to extract allowed changes checks from the old
validation class.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: fix _fetch_export to distinguish between clusters

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: move export ganesha conf translation into caller

This belongs in the (one) caller, not the helper.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: name nfs cephfs client key 'nfs.{cluster_id}.{export_id}'

Better than '{cluster_id}{export_id}', which is confusing and ambiguous.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: add --addr to 'nfs export create'

Ganesha exports can have multiple client blocks with addresses and
access modes/squash behavior.  This cannot easily be exposed via a CLI.
However, providing no client address restriction is pretty useful to users
since it allows full access to the mount point with no restrictions.

Add an --addr argument that creates a single client block.  Set the top
export-level access_type to 'none' and squash to 'none' in this case.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: add --squash to 'nfs export create'

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs/export_utils: include false but non-None items in config

For example, attr_expiration_time = 0 is the default. Include this since
it is not None.

Signed-off-by: Sage Weil <sage@newdream.net>

vstart.sh: enable nfs module

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: nfs: drop attr_expiration_time from top-level config

This is added on a per-export basis with the same default of 0.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: remove Dir_Chunk = 0

This breaks RGW exports, and is likely to be beneficial for libcephfs.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #41905 from ivancich/wip-improve-cls-rgw-tracing

rgw: clean-up logging of function entering to make thorough and consistent

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

rgw: clean-up logging of function entering to make thorough and consistent

This provides more thorough and consistent function tracing in CLS/RGW
when logging is set to 10 or higher.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #41903 from liewegas/update-rook-client

rook-client-python: update to update-june-21

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41835 from TRYTOBE8TME/wip-rgw-keycloak-failure-fix

qa/tasks: Keycloak failure fix

Reviewed-by: Ali Maredia <amaredia@redhat.com>

mgr/rook: comment out osd creation functions

This commit comments out the OSD creation functions in rook_cluster.py
and module.py, since the submodule update has broken them.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

mgr/rook: fix some mypy typing errors in rook_cluster.py

This commit fixes some errors caught by mypy in rook_cluster.py. Most of the
errors were caused by the update of the rook-client-python submodule in a previous
commit.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

mgr/rook: pass zone attribute to CephObjectStore CR when creating rgw

This commit passes the zone attribute to the CephObjectStore CR when
creating a RGW instance using the Rook Orchestrator backend:
`ceph orch apply rgw <rgw-name> --realm=<realm-name> --zone=<zone-name>`

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

Merge pull request #41856 from rhcs-dashboard/maintenance-bug-fix

mgr/dashboard: Fix 500 error while exiting out of maintenance

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>

Merge pull request #41694 from jmolmo/kcli_cephadm_doc

doc: Add kcli utilization for development environments

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41848 from xxhdx1985126/wip-errorator-parallel_for_each

crimson: errorator parallel_for_each

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41894 from tchaikov/wip-crimson-sigint

crimson/{osd,store_nbd}: handle SIGINT

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #41895 from ceph/wip-cacephcom

MIRRORS: Add ca.ceph.com

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix 500 error while exiting out of maintenance

When you add a host in maintenance mode and then exit the maintenance
mode, a 500 server error will popup which will interrupt the whole
exit maintenance process and leave the host in an unknown/offline state.
It happened when I was setting the status of the host through the
HostSpec(). With this change, I am using the enter_maintenance api of
the orch to enable the maintenance.

Fixes: https://tracker.ceph.com/issues/51218
Signed-off-by: Nizamudeen A <nia@redhat.com>

rook-client-python: update to update-june-21

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #41890 into master

* refs/pull/41890/head:
doc/cephadm: removing "Octopus" from procedure

Reviewed-by: Sage Weil <sage@redhat.com>

MIRRORS: Add ca.ceph.com

Signed-off-by: David Galloway <dgallowa@redhat.com>

crimson/osd: use stop_signal from seastar

and disable app_cfg.auto_handle_sigint_sigterm, otherwise app template
handles SIGINT and SIGTERM by itself, and calls app.stop(). but we don't
use this mechinary at all. we use seastar::defer() instead of
seastar::at_exit() for doing graceful shutdown and cleanup.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd: update example usage in comment

--total-device-size is not supported any more.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd: add graceful shutdown support

we could have more sophisticate mechinary for interrupting fio job,
but so far it is able to stop itself if it idle by handling ctrl-C.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/seastore/nbd: destruct the store before create

Otherwise the store will register the conflicting metrics and result in
double_registration.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc/cephadm: removing "Octopus" from procedure

This PR removes "Octopus" from the curl-based installation
procedure.

After we moved on to Pacific, referring to Octopus looks wrong.
It looks wrong because it now is wrong.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41882 from tchaikov/wip-crimson-int-safty

crimson/osd: guard non-pg-op handling with with_sequencer()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41885 from tchaikov/wip-crimson-os-cleanups

crimson/os: cleanups and reformat

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/osd: reindent

for less indent, hence better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: wrap line at 80

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: guard non-pg-op handling with with_sequencer()

because we should only ensure the ordering of the requests touching
the objects, the other requests like pgls should not be ordered along
with them. so as the second step, guard the non-pg-op handling with
with_sequencer().

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: use reference for loop variable

for better performance, also silences the warning like:

../src/crimson/os/seastore/random_block_manager/nvme_manager.cc:444:23: warning: loop variable ‘b’ creates a copy from type ‘const crimson::os::seastore::rbm_alloc_delta_t’ [-Wrange-loop-construct]
  444 |       for (const auto b : alloc_blocks) {
      |                       ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41859 from sebastian-philipp/mypy-constrains.txt

global,tox.ini: add mypy-constrains.txt

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

crimson/os: return seastar::now() in "finally()" block

so finally() is able to identify the return is a future, and discard it
manually.
otherwise the return value will be discarded even the future is marked
[[nodiscard]], hence the C++ compiler warns.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: remove unnecessary now()

the previous continuation in the chain already returns a future, no need
to hook up another now().

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: remove unnecessary parentheses

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: fix the indent

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not capture unused variables

Signed-off-by: Kefu Chai <kchai@redhat.com>

global,tox.ini: add mypy-constrains.txt

let's avoid getting new versions of those packages by accident.
Unfortunately this means we have to manually update those
packages regurarly.

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

crimson/osd: extract with_sequencer() method out

because we should only ensure the ordering of the requests touching
the objects, the other requests like pgls should not be ordered along
with them, so as the first step, let's extract with_sequencer() method
out, so we can differentiate these two cases with minimal repeatings.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41881 from tchaikov/wip-crimson-alien

crimson: adapt to the new alien API

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: adapt to the new alien API

since seastar now deprecates the old alien::submit_to() API, and
replaced it with a variant which requires a new parameter of "alien"
instance, let's pick it up, to amortize the cost to catch up with
the seastar master HEAD.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: add test case for errorator parallel_for_each

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson: add parallel_for_each to errorator

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #41825 from tchaikov/wip-avl-cleanup

os/bluestore/AvlAllocator: specialize _block_picker() and cleanups

Reviewed-by: Igor Fedotov <ifedotov@suse,com>