mgr/volumes: Move ganesha common config to vstart

This is a preparatoy patch before calling orchestrator for nfs cluster
deployment. All the ganesha config is moved to vstart. Keyring creation is also
taken care by vstart.

The volumes fs nfs cluster create interface does the following things:
1) Create a common recovery pool for all ganesha clusters. Each cluster will
   have their own namespace.
2) Create an empty rados conf object named 'conf-nfs' for saving all export
   urls.

Call to orch interface will be done in future patch.

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Fix mypy errors

This patch fixes the following mypy errors:
volumes/module.py:9: note: In module imported here,
volumes/__init__.py:2: note: ... from here:
volumes/fs/nfs.py: note: In member "validate_path" of class "CephFSFSal":
volumes/fs/nfs.py:80: error: Name 're' is not defined
volumes/fs/nfs.py: note: In member "create_path" of class "CephFSFSal":
volumes/fs/nfs.py:83: error: Name 'CephFS' is not defined
volumes/fs/nfs.py: note: In member "_persist_daemon_configuration" of class "GaneshaConf":
volumes/fs/nfs.py:259: error: Need type annotation for 'daemon_map' (hint: "daemon_map: Dict[<type>, <type>] = ...")
volumes/fs/nfs.py: note: In member "create_instance" of class "NFSConfig":
volumes/fs/nfs.py:386: error: Incompatible types in assignment (expression has type "GaneshaConf", variable has type "str")
volumes/fs/nfs.py: note: In member "create_export" of class "NFSConfig":
volumes/fs/nfs.py:389: error: "str" has no attribute "create_export"
volumes/fs/nfs.py: note: In member "delete_export" of class "NFSConfig":
volumes/fs/nfs.py:404: error: "str" has no attribute "has_export"
volumes/fs/nfs.py:407: error: "str" has no attribute "remove_export"
volumes/fs/nfs.py:408: error: "str" has no attribute "reload_daemons"
volumes/__init__.py:2: note: In module imported here:
volumes/module.py: note: In member "_cmd_fs_nfs_export_create" of class "Module":
volumes/module.py:406: error: "str" has no attribute "check_fsal_valid"
volumes/module.py:407: error: "str" has no attribute "create_instance"
volumes/module.py:408: error: "str" has no attribute "create_export"
volumes/module.py: note: In member "_cmd_fs_nfs_export_delete" of class "Module":
volumes/module.py:412: error: "str" has no attribute "delete_export"
volumes/module.py: note: In member "_cmd_fs_nfs_cluster_create" of class "Module":
volumes/module.py:415: error: Incompatible types in assignment (expression has type "NFSConfig", variable has type "str")
volumes/module.py:416: error: "str" has no attribute "create_nfs_cluster"

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Update the export class to remove unecessary config options

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Remove dependency on dashboard ganesha module

Instead of importing ganesha module, necessary classes are directly used in
volumes nfs module.

Signed-off-by: Varsha Rao <varao@redhat.com>

vstart: Use random port instead of default Ganesha port

Signed-off-by: Varsha Rao <varao@redhat.com>

vstart: Use "NFS" instead "GANESHA" for starting nfs ganesha clusters

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Improve readability of ganesha common config

Signed-off-by: Varsha Rao <varao@redhat.com>

vstart: Add note about mounting cephfs exports

Signed-off-by: Varsha Rao <varao@redhat.com>

src/vstart: Set CEPH_CONF environment variables

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/fs: Update default ganesha conf options

Signed-off-by: Varsha Rao <varao@redhat.com>

doc: Add document about fs nfs interface to create CephFS exports

Signed-off-by: Varsha Rao <varao@redhat.com>

vstart: Update fs nfs export create command

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Update nfs export create and delete command

Remove fill_keys function and add key to nfsconfig object.
Export create and delete methods that no longer require ganeshaconf object to
be passed.

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Remove create_rados_pool method

Pool is created by cluster command.

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Changes to nfs export delete and create

Fixes: https://tracker.ceph.com/issues/44193
Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: While creating nfs-ganesha user update its cap

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Update caps for new user created and add it's key to keyring

Fixes: https://tracker.ceph.com/issues/44193
Signed-off-by: Varsha Rao <varao@redhat.com>

vstart: Update vstart according to cluster create command

Fixes: https://tracker.ceph.com/issues/44193
Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Add Ganesha common config

Fixes: https://tracker.ceph.com/issues/44193
Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Create NFSConfig class

Move the ganesha config functions into this class.

Fixes: https://tracker.ceph.com/issues/44193
Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Add command to create nfs-ganesha clusters

ceph fs nfs cluster create <cluster_id> [--size=1]

cluster_id: Name of the nfs cluster

This command creates a common recovery pool for all Ganesha daemons, creates
new user and nfs cluster. Orchestrator needs to be enabled.

Fixes: https://tracker.ceph.com/issues/44193
Signed-off-by: Varsha Rao <varao@redhat.com>

mgr: Create pool for nfs ganesha recovery

Fixes: https://tracker.ceph.com/issues/44193
Signed-off-by: Varsha Rao <varao@redhat.com>

mgr: Add "ceph fs nfs delete <export_id>" to delete exports

Fixes: https://tracker.ceph.com/issues/44193
Signed-off-by: Varsha Rao <varao@redhat.com>

mgr: Add command to create exports for nfs-ganesha

'ceph fs nfs create': This command creates export objects for nfs-ganesha.
vstart ganesha daemon fetches the export block from rados object.

Fixes: https://tracker.ceph.com/issues/44193
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #34229 from Yan-waller/wip-walle-fixsparsereadlength

osd/PrimaryLogPG: fix SPARSE_READ stat

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34342 from ideepika/fixes-44862

mon: calculate min_size on osd pool set size

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34219 from yanghonggang/bluefs-tool

os/bluestore: Don't pollute old journal when add new device

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #34143 from tchaikov/wip-mgr-disable-dne-module

mon/MgrMonitor: show different error message when disabling a dne module

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #34366 from SUSE/wip-mgr-fix-python-traceback

mgr/PyModule: fix missing tracebacks in handle_pyerror()

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34337 from majianpeng/throttle-remove-lock

common/Throttle: Don't lock for atomic type update.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34381 from rhcs-dashboard/fix-44721-master

rpm: add python3-saml as install dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34409 from adamemerson/wip-namespace-osd

osd: build without `using namespace` declarations in headers

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34460 from majianpeng/cmakefile-fix

cmake: remove duplicated code.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34451 from tchaikov/wip-standalone-pgid

qa/standalone/scrub: s/$(pgid)/${pgid}/

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34245 from rzarzynski/wip-bug-24995

mgr: synchronize ClusterState's health and mon_status.

Reviewed-by: Tim Serong <tserong@suse.com>

cmake: remove duplicated code.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #34368 from majianpeng/msg-remove-unsued-code

msg, common/Throttle: remove unsued code.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34369 from smithfarm/wip-43895

cmake: really stop at top of source code tree

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34383 from matthewoliver/older-tox

tox: Fix the tox.ini's to support older versions of tox

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34427 from smithfarm/wip-44964

spec: address some warnings raised by RPM 4.15.1

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33945 from rishabh-d-dave/fs-qa-vstart_runner.LocalRemote.sh

qa/vstart_runner: update vstart_runner.LocalRemote.sh

Reviewed-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #34456 from ceph/wip-crontab-master

qa/tests: removed ceph-deploy from master and octopus (ceph-deploy is…

qa/tests: removed ceph-deploy from master and octopus (ceph-deploy isn't written to run on el8)

Signed-off-by: yuriw <yuri.weinstein@gmail.com>

Merge pull request #34439 from jschmid1/osdspec_encrypted_docs

docs/cephadm: <encrypted> is a global flag

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34053 from matthewoliver/iscsi-cephadm

cephadm: Add ceph-iscsi

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34357 from mgfritch/mgr-orch-test-warn

mgr/orch: fix python3 DeprecationWarning

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/standalone/scrub: s/$(pgid)/${pgid}/

to address the test failures like
```
2020-04-07T15:44:58.693 INFO:tasks.workunit.client.0.smithi049.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_failed:  ceph pg dump
pgs
2020-04-07T15:44:58.694 INFO:tasks.workunit.client.0.smithi049.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_failed:  pgid
2020-04-07T15:44:58.694 INFO:tasks.workunit.client.0.smithi049.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh: line 498: pgid: command not found
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: add python3-saml as install dependency

`python.*-saml` is required for Ceph-Dashboard SSO support (optional
feature).

Fixes: https://tracker.ceph.com/issues/44721
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #34396 from tspmelo/wip-npm-hacking

mgr/dashboard: Update HACKING.rst

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34412 from jan--f/c-v-fix-simple-scan-log-string

devices/simple/scan: Fix string in log statement

doc/cephadm: <encrypted> is a global flag

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34406 from liewegas/fix-cephadm-upgrade-start

qa/suites/rados/cephadm/upgrade: start from v15.2.0

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34310 from badone/wip-restful-api-dont-pass-empty-dict-to-get

qa/suites/rados/rest: don't pass empty dict as data arg

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #34302 from badone/wip-specify-rule-for-pool-creation

Wip specify rule for pool creation

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

ceph.spec.in: remove extra tokens at end of %endif directive

RPM 4.15.1 warns: extra tokens at the end of %endif directive

Heed the warning.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: version an Obsoletes

RPM 4.15.1 warns: It's not recommended to have unversioned Obsoletes.

Heed the warning.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

osd: Build ceph-osd without using namespace declarations in headers

This is part of a series of commits to clean up using namespace at top
level in headers.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

{common, rgw}: Build ceph-osd without namespace pollution in headers

This is part of a series of commits to clean up using namespace at top
level in headers.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

include/ceph_assert: qualify __assert_warn call in assert_warn macro

A macro never knows where it will be expanded.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

perfglue: Build ceph-osd without using namespace declarations in headers

This is part of a series of commits to clean up using namespace at top
level in headers.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls: Build ceph-osd without using namespace declarations in headers

This is part of a series of commits to clean up using namespace at top
level in headers.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #34239 from p-se/wip-pse-fix-false-root-vol-full-alert

monitoring: root volume full alert fires false positives

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34240 from krig/grafana-dashboards-fixes

mgr/dashboard: Repair broken grafana panels

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

devices/simple/scan: Fix string in log statement

A multiline string enclosed by () gained a comma, making it into a
tuple, which has no format() method.

Fixes: https://tracker.ceph.com/issues/44949
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

mgr/dashboard: Update HACKING.rst

This will update information regarding npm.

Fixes: https://tracker.ceph.com/issues/44927
Signed-off-by: Tiago Melo <tmelo@suse.com>

common: move to_string and to_integer overloads into root namespace

Since that's where the type is defined, that's where overloads taking
it should be defined.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #34374 from sebastian-philipp/mgr-fix-tox-cephadm

pybind/mgr: Fix `run_tox.sh mgr -- cephadm`

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34269 from yuvalif/fix_cls_queue_logs

cls/queue: remove error message when queue is not initialized

Merge pull request #33985 from zhengchengyao/clone_metadata

librbd: children should inherit parent's stripe

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34072 from shangdehao1/fix_nfs_bug

qa/tasks/qemu: fix nfs setup and teardown bug in qemu task

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34371 from smithfarm/wip-43896

doc/releases/nautilus: restart OSDs to make them bind to v2 addr

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #31480 from ukernel/wip-20

client: auto reconnect after blacklisted

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #33915 from lxbsz/client_assert_failedddd

Client: fix Finisher assert failure

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #34386 from gregsfortytwo/wip-33279-revert-sudo

Wip 33279 revert sudo

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #34110 from ukernel/wip-44680

mds: don't assert empty io context list when shutting down

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #34385 into master

* refs/pull/34385/head:
doc/dev/cephadm: a few notes on developing with cephadm

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #34281 from vshankar/wip-44677

mgr: force purge normal ceph entities from service map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34306 from ukernel/wip-44771

ceph-fuse: don't get mount options from /etc/fstab when doing remount

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #34296 into master

* refs/pull/34296/head:
qa/suites/rados/cephadm/smoke-roleless: add smoke test
qa/tasks/cephadm: add 'roleless' mode

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34384 into master

* refs/pull/34384/head:
cephadm: create /var/run/ceph dir via unit.run, not unit file

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/suites/rados/cephadm/upgrade: start from v15.2.0

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #34405 from ceph/wip-yuriw-cron-master2

qa/tests: fixed typo

qa/tests: fixed typo

Signed-off-by: yuriw <yuri.weinstein@gmail.com>

Merge pull request #34404 from ceph/wip-yuriw-cron-master

qa/tests: added client upgrades tests for octopus, removed mimic-x/ma…

qa/tests: added client upgrades tests for octopus, removed mimic-x/masted as not needed

Signed-off-by: Yuri Weinstein <yweinstein@redhat.com>

doc/dev/cephadm: a few notes on developing with cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33502 from yison/rwl-image-writeback-cache-seq3

rbd/cache: Replicated Write Log core codes part 3

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34362 from tchaikov/wip-rbd-with-pmem

cmake, librbd: fix build with pmem and cleanups

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34332 from cyx1231st/wip-seastar-messenger-lossy-peer

crimson/net: enable features for lossy peer connections of heartbeat

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34319 from clyso/patch-2

doc/mgr/orchestrator: update cephadm shell proposed alias entry

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34318 from clyso/patch-1

doc/mgr/orchestrator: add path to ssh-copy-id instructions

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34367 from sebastian-philipp/codeowners-cephadm-update

CODEOWNERS: update cephadm paths

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34390 from tchaikov/wip-github-codeowners

.github/CODEOWNERS: add ceph/crimson

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #34325 from tspmelo/wip-npm-update-20-03

mgr/dashboard: Update all npm packages

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34377 from votdev/issue_44914_rgw_debug_info

mgr/dashboard: Add more debug information to Dashboard RGW backend

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #34333 from changchengx/mail_org_update

mailmap: update mail org relationship

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/mgr/orchestrator: update cephadm shell proposed alias entry

prior change i get following error when using alias as "ceph -v":
INFO:cephadm:Inferring fsid ecfdb924-7341-11ea-943c-020100010027
INFO:cephadm:Using recent ceph image ceph/ceph:v15
/usr/bin/docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: "/usr/bin/ceph -v": stat /usr/bin/ceph -v: no such file or directory": unknown.
after:
INFO:cephadm:Inferring fsid ecfdb924-7341-11ea-943c-020100010027
INFO:cephadm:Using recent ceph image ceph/ceph:v15
ceph version 15.2.0 (dc6a0b5) octopus (rc)

Signed-off-by: Tobias Fischer <tobias.fischer@clyso.com>

doc/mgr/orchestrator: add path to ssh-copy-id instructions

by default ssh key will be placed under /etc/ceph - so it should be included in examples

Signed-off-by: Tobias Fischer <tobias.fischer@clyso.com>

Merge pull request #34227 from p-se/wip-pse-fix-alert-space-prediction

monitoring: alert for prediction of disk and pool fill up broken

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

.github/CODEOWNERS: add ceph/crimson

so we can add reviewers in a more efficient way for crimson related
changes

Signed-off-by: Kefu Chai <kchai@redhat.com>