common/cmdparse: fix CephBool validation for tell commands

The tell/asok validation uses a different validation path; make it work
for CephBool arguments.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: fix 'nfs export create' argument order

Put path before --readonly so that it can still be positional.

Signed-off-by: Sage Weil <sage@newdream.net>

common/cmdparse: emit proper json

Instead of '"req": "false"', emit '"req": false'.  Same for conditional.

Luckily, the clients don't really care about this change, as
ceph_argparse.py argdesc interpets the JSON like so:

            self.req = req in (True, 'True', 'true')
            self.positional = positional in (True, 'True', 'true')

Clean up command definitions to use lowercase 'false', but tolerate
both for backward compat during upgrade and to tolerate future errors.

Signed-off-by: Sage Weil <sage@newdream.net>

mon/MonCommands: add -- seperator to example

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephfs/test_nfs: fix export create test

Everything after --readonly is non-positional.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr: make mgr commands compat with pre-quincy mon

If the mon is not yet quincy, do not share the quincy-ism of
'positional=false'.

Signed-off-by: Sage Weil <sage@newdream.net>

doc/_ext/ceph_commands: handle non-positional args in docs

Signed-off-by: Sage Weil <sage@newdream.net>

mgr: fix reweight-by-utilization cephbool flag

This was broken, since it was trying to get a CephChoices as a bool.

Signed-off-by: Sage Weil <sage@newdream.net>

mon/MonCommands: convert some CephChoices to CephBool

These are old bool options that never got converted to CephBool.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/k8sevents: fix help strings

Signed-off-by: Sage Weil <sage@newdream.net>

pybind/mgr/mgr_module: fix help desc formatting

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/orchestrator: clean up 'orch {daemon add,apply} rgw' args

Make placement the only optional positional.  This means that the "usual"
'orch orch apply <whatever> 3' will do placement=3.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/orchestrator: add end_positional to a few methods

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/orchestrator: reformat a few methods

Signed-off-by: Sage Weil <sage@newdream.net>

pybind/ceph_argparse: stop parsing when we run out of positional args

If we encouter an arg that is not a named flag/arg, and the next
item in the command description is non-positional, then raise an
'unexpected argument' exception.

Signed-off-by: Sage Weil <sage@newdream.net>

pybind/ceph_argparse: remove dead code

Signed-off-by: Sage Weil <sage@newdream.net>

pybind/mgr/mgr_module: infer non-positional args

Once we have an Optional[bool], we can always transition to non-positional,
since we never have a non-optional bool.  Same goes for the 'format'
arg.

Signed-off-by: Sage Weil <sage@newdream.net>

pybind/mgr/mgr_module: add separator for non-positional args

Signed-off-by: Sage Weil <sage@newdream.net>

command/cmdparse: use -- to separate positional from non-positional args

In a command definition, separate the non-positional args with "--".

Signed-off-by: Sage Weil <sage@newdream.net>

pybind/ceph_argparse: adjust help text for non-positional args

If an arg is non-positional, always show it as

  [--arg-name <value>]

(All non-positional args are optional.)

Signed-off-by: Sage Weil <sage@newdream.net>

pybind/ceph_argparse: track a 'positional' property on cli args

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #41382 from tchaikov/wip-doc-msgr-fsm-diagram

doc/dev/msgr2: add state machine diagrams

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41408 from zdover23/wip-doc-security-2021-05-19-fourth-item

doc/security: updating fourth listitem

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41407 from zdover23/wip-doc-security-2021-05-19-second-item

Wip doc security 2021 05 19 second item

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #41358 into master

* refs/pull/41358/head:
mds: add heartbeat_reset() in start_files_to_reover() to avoid timedout issue

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41338 into master

* refs/pull/41338/head:
mds: reset code after cpu_profiler
cpu_profiler: fix asok command crash

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41084 into master

* refs/pull/41084/head:
test: test to verify dir path removal when no mirror daemons are running
pybind/mirroring: advance state machine from stalled state
pybind/mirroring: start from correct state during policy init

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/security: updating fourth item

This PR makes minor changes (nitpicking, really)
to make the sentence a little easier to read.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/security: enriching second point

This PR breaks a long sentence into two shorter
sentences and improves the language.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41411 from zdover23/wip-doc-security-2021-05-19-eighth-item

doc/security: enriching eighth listitem

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41437 from tchaikov/wip-crimson-logging

crimson/os/seastore/logging: remove comma

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41070 from sunnyku/wip-rbd-50522

rbd: fix default pool handling for nbd map/unmap

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #41340 from tchaikov/wip-ceph-allocator

librbd: do not use ceph::allocator<>

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/security: enriching eighth listitem

This PR improves the language in the eighth
listitem in the Vulnerability Management Process.

(This one's pretty nitpicky.)

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #40328 from rhcs-dashboard/fix-49262-master

mgr/dashboard: provide the service events when showing a service in the UI

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore/logging: remove comma

to address the FTBFS of the release builds.

fix a regression introduced by c39d64d7bbd7d7cf1d476a44f60c70edce55f3d1

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41396 from tchaikov/wip-crimson-metrics

crimson/os: replace perfcounter with seastar::metrics

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #41258 from ronen-fr/wip-ronenf-repair-stat

osd/scrub: separate between PG state flags and internal scrubber operation

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41264 from mgfritch/cephadm-sysctl-osd

cephadm: apply sysctl settings

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #41368 from idryomov/wip-monc-rotating-timeout

mon/MonClient: tolerate a rotating key that is slightly out of date

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #41277 from david-caro/consider_dev_root_mounted

ceph-volume: Consider /dev/root as mounted

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #41287 from adk3798/daemon-for-service-warn

mgr/cephadm: warn users if they try to pass a daemon to 'orch rm'

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge pull request #41356 from tchaikov/wip-options-mgr-mon

common/options: extract mgr and mon options out

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #41373 from myoungwon/wip-fix-50806

 osd: fix wrong input when calling recover_object()

Reviewed-by: Samuel Just <sjust@redhat.com>

test/crimson: remove unnecessary local variable

to return the return value of "make_unique<>()" is more readable, and more
consistent with other part of this source file.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson/seastore: teardown in reactor

otherwise, we rely on the destructor of TMTestState to teardown the
fixuture created in TMTestState::_init(), but TMTestState::_init() is
called in reactor. the objects like seastar::metric_groups are
supposed to be destroyed on the same thread where they are created.
because they use thread local storage of storing persisting their status.
if we destroy objects like seastar::metric_groups on different reactor
or thread where they are created, we would have memory leak and
unexpected behavior.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: drop seastore_perf_counters.h

it is not used anymore after the seastar::metrics migration

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd: remove PerfService

since PerfCounter is not used in seastore, there is no need to keep an
instance of PerfService.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not create PerfService for SeaStore

because we are replacing PerfCounter with seastart::metrics in crimson,
and the former has already a sharded service builtin in seastar, there
is no need to create PerfService or its counterpart for SeaStore
anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: move ctor of SeaStore into .cc

so the ctor and dtor are colocated in the same file. for better
readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: replace perfcounter with metrics in SegmentCleaner

for better support of m:n mapping, and better maintainability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: register metrics() in TransactionManager's ctor

this change should have been a part of
2356da856784e78750ba7f3b381b31b1c8ad5672

also do not include the unused header file

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: move ctor of SegmentCleaner into .cc

will add more stuff into it. and presumably, smaller header file, faster
compilation.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41423 from athanatos/sjust/wip-lba-merge-fix

crimson/.../lba_btree_node_impl: handle relative addr in merge

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #41392 from tchaikov/wip-librados

include/rados/librados.h: use UINT64_C to define constants

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41409 from zdover23/wip-doc-security-2021-05-19-fifth-item

doc/security: updating fifth listitem

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41410 from zdover23/wip-doc-security-2021-05-19-seventh-item

doc/security: enrich seventh listitem

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41424 from athanatos/sjust/wip-interruptor-unittest-name

test/crimson/CMakeLists.txt: fix name of unittest-interruptible-future

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@360.cn>

Merge pull request #41425 from athanatos/sjust/wip-seastore-build

crimson: fix misc build issues

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd/tm_driver: remove perfcounter argument

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/logging: fix log() argument

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/CMakeLists.txt: fix name of unittest-interruptible-future

unittest-interruptible-future matches the other unit tests here.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../lba_btree_node_impl: handle relative addr in merge

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #40836 from kamoltat/wip-autoscaler-new-alg-turn-on-off

pybind/mgr/pg_autoscaler: Added autoscale-profile feature

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

pybind/mgr/pg_autoscaler: Added autoscale-profile feature

The autoscaler by default will start out each pool with minimal
pgs and `scale-up` the pgs when there is more usage in each pool.

Users can now use the commands:

`osd pool set autoscale-profile scale-down` to make the pools
start out with a full complement of pgs and only `scale-down`
when usage ratio across the pools are not even.

`osd pool set autoscale-profile scale-up` (by default) to make the pools
start out with minimal pgs and `scale-up` the pgs when there
is more usage in each pool.

Edited KVMonitor.cc file to make the `autoscale_profile` variable
persistent.

Edited tests/test_cal_final_pg_target.py so that it takes into account
the new `profile` argument when calling cal_final_pg_target(). Also,
added some new test cases for when profile is `scale-up`

Renamed tests/test_autoscaler.py to a more appropriate name:
tests/test_cal_ratio.py

Signed-off-by: Kamoltat <ksirivad@redhat.com>

doc/security: enrich seventh listitem

This PR improves the language of the seventh listitem
in the Vulnerability Management Process in the security
documentation.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41406 from zdover23/wip-doc-security-2021-05-19-first-item

doc/security: removing pleonasm

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/security: updating fifth listitem

This PR enriches the text in the fifth listitem
of the Vulnerability Management Process.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41391 from tchaikov/wip-libcephfs

libcephfs: include <sys/time.h>

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #41359 from tchaikov/wip-cmake-cleanup

cmake: cleanups related to file(GLOB_RECURSE..) call

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc/security: removing pleonasm

s/three business days or less/three business days/

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41355 from tchaikov/wip-doc-rgw-conf

doc/radosgw: use confval directive to define options

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41399 from idryomov/wip-mrun-if-posix

mrun: avoid == in [ command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41393 from rosinL/fix-50835

common/crc32c_aarch64: fix crc32c unittest failed on aarch64

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/radosgw: use confval directive to define options

less repeating this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #41388 into master

* refs/pull/41388/head:
cephadm: write config files as utf-8

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #41286 into master

* refs/pull/41286/head:
qa/suites/orch/rook: disable centos for now
qa/suites/orch/rook/smoke: initial smoke suite
qa/tasks/rook: ROOK_HOSTPATH_REQUIRES_PRIVILEGED=true on centos
qa/tasks/rook: simplify shutdown
qa/tasks/rook: archive logs
qa/tasks/rook: more orderly cluster teardown
qa/tasks/rook: deploy ceph via rook on top of kubernetes
qa/tasks/kubeadm: install kubernetes with kubeadm
qa/suites: move rados/cephadm -> orch/cephadm; symlink
qa/tasks/cephadm: add whitespace between functions
qa/tasks/cephadm: clean up ctx.manager setup

Reviewed-by: Sébastien Han <seb@redhat.com>

Merge PR #41370 into master

* refs/pull/41370/head:
mgr/cephadm: pin haproxy image to 2.3.z

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #41345 from rhcs-dashboard/fix-CVE-2021-3509-master

mgr/dashboard: fix cookie injection issue (CVE-2021-3509)

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mrun: avoid == in [ command

== operator is not supported in POSIX sh (e.g. dash):

  ./mrun: 24: [: unexpected operator

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #41104 from rhcs-dashboard/fix-ESOCKETTIMEDOUT-e2e-failure

mgr/dashboard: fix ESOCKETTIMEDOUT E2E failure

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41273 from rhcs-dashboard/update-frontend-deps

mgr/dashboard: update frontend deps due to security vulnerabilities

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

common/crc32c_aarch64: fix crc32c unittest failed on aarch64

On centos 8.2 for aarch64 with gcc 8.3, the complier will use
register v0 conflicting with the register v0 be usded in inline
asm code. Adding the related registers into clobber list to inform
complier avoiding the confict.

Fixes: https://tracker.ceph.com/issues/50835
Signed-off-by: luo rixin <luorixin@huawei.com>

cephadm: write config files as utf-8

If we don't specify the encoding, python may use something like 'latin-1',
which cannot handle unicode characters.

Signed-off-by: Sage Weil <sage@newdream.net>

include/rados/librados.h: use UINT64_C to define constants

UINT64_C add the appropriate suffix to literals, this is more readable
than casting the literal to the expected type.

Signed-off-by: Kefu Chai <kchai@redhat.com>

libcephfs: include <sys/time.h>

header files should be self-contained. we are using `timeval` in the header,
but failed to include the header defining it.

this change addresses the build failures on Darwin like:

int ceph_lutimes(struct ceph_mount_info *cmount, const char *path, struct timeval times[2]);
                                                                          ^
src/include/cephfs/libcephfs.h:1080:78: error: array has incomplete element type 'struct timeval'
int ceph_futimes(struct ceph_mount_info *cmount, int fd, struct timeval times[2]);
                                                                             ^
src/include/cephfs/libcephfs.h:1080:65: note: forward declaration of 'struct timeval'
int ceph_futimes(struct ceph_mount_info *cmount, int fd, struct timeval times[2]);
                                                                ^
3 errors generated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41390 from batrick/i50865

doc: use sphinx .dot include

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #41357 into master

* refs/pull/41357/head:
mds: do not assert when receiving a unknow metric type

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #41347 into master

* refs/pull/41347/head:
mon,doc: deprecate CephFS min_compat_client
doc: add alternate_name cephfs feature to table

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #41332 into master

* refs/pull/41332/head:
mds: place the journaler pointer under the mds_lock

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41171 into master

* refs/pull/41171/head:
test: disable mirroring module for certain tests

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41097 into master

* refs/pull/41097/head:
cephfs-mirror: allow connecting to local cluster using mon address

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sébastien Han <seb@redhat.com>

Merge PR #40885 into master

* refs/pull/40885/head:
doc: document cephfs-mirror configuration options
cephfs-mirror: use sensible mount timeout when mounting local/remote fs
test: add tests for settting mount timeout
pybind/cephfs: add interface to set mount timeout
libcephfs: add interface to set mount timeout

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41371 into master

* refs/pull/41371/head:
mds/Mutation: initialize MutationImpl object's initiated_at

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/suites/orch/rook: disable centos for now

We are hitting an selinux denial:

type=AVC msg=audit(1621353663.595:6016): avc: denied { write } for pid=28812 comm="cephcsi" name="pids.max" dev="cgroup" ino=2933 scontext=system_u:system_r:container_t:s0:c43,c177 tcontext=system_u:object_r:cgroup_t:s0 tclass=file permissive=1

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/orch/rook/smoke: initial smoke suite

- os: 18.04, 20.04, centos, centos+kubic
- nodes: 1 or 3
- rook: 1.6.2 or master
- k8s: 1.21 always (for now)
- net: calico always (flannel + centos is wonky)
- workload: none, rados bench

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rook: ROOK_HOSTPATH_REQUIRES_PRIVILEGED=true on centos

This fixes OSD creation on centos.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rook: simplify shutdown

For some reason deleting common.yaml sometimes fails.  Not really
sure why, but since we will tear down kubernetes anyway this
cleanup isn't really needed.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rook: archive logs

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rook: more orderly cluster teardown

Signed-off-by: Sage Weil <sage@newdream.net>