test: remove quirky argparse case

This relied on a behaviour where positional
arguments could be omitted if the subsequent
argument was of a different type.

This was pretty weird, and in any case the reweight-by-utilization
command is likely to go away soon.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr,mon: use CephChoices for confirmation flags

...so that arg parsing can rely on strings satisfying
a "doesn't start with --" convention.

Signed-off-by: John Spray <john.spray@redhat.com>

test: add cases for CLI's --key=val style

Signed-off-by: John Spray <john.spray@redhat.com>

mon: remove dead "cluster_snap" command

This had existed in a disabled state (by having an empty
string for the cli/rest field in the command definition)
for a long time.  Now that that field is gone, we don't
have a concept of "disabled" commands any more, so
let's just clean up this loose end.

Signed-off-by: John Spray <john.spray@redhat.com>

pybind: enable --keyword=arguments in ceph_argparse

This is a simple implementation that treats anything
that matches the "--X=Y" pattern as separate from
positional arguments.

This works well for optional arguments.  Mandatory
arguments still need to be specified positionally,
or the parsing code will think the command's
argument description has not been satisfied.

Signed-off-by: John Spray <john.spray@redhat.com>

ceph.in: some cleanups

Add some docstrings to functions, remove some
dead code and places where e.g. dicts were used
unncessary.

Signed-off-by: John Spray <john.spray@redhat.com>

ceph.in: misc cleanups

Get this pyflakes-clean ahead of making
changes for keyword arguments.

Signed-off-by: John Spray <john.spray@redhat.com>

common: remove unused 'avail' field from commands

This was introduced for the now-removed ceph-rest-api
gateway.  It enabled limiting certain commands
to be CLI-only or rest-only, but in practice almost
everything just said "cli,rest" here.

Now that ceph-rest-api is gone, let's remove this
field.

The CLI client code already tolerated the absence
of this field, so older CLI clients will not mind.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: fix help string for `osd crush rule create-replicated`

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #24616 from tchaikov/wip-wip-librados-3

librados: split C++ and C APIs into different source files

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #24555 into master

* refs/pull/24555/head:
doc: document optional timeout argument for `cache drop`
mds: make timeout parameter optional for "cache drop"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24747 into master

* refs/pull/24747/head:
client: add new routine to get fscid from a ceph_mount

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24868 from liewegas/wip-doc-fix

doc/rados/operations/crush-map-edits: fix 'take' syntax

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rpm,deb: package libceph-common.so.* not libceph-common.so*

be more explicit on what we are packaging. because only
libceph-common.so.${soversion} will be packaged, since libceph-common.so
won't be installed by cmake anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not create symlink for libceph-common.so.*

and bump up its soversion, so it won't conflict with libceph-common.so.0
offered by librados2.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: add librados3 to exclude list of pre-nautilus install tasks

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: s/librados2/librados3/

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: s/librados2/librados3/

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: s/librados2/librados3/

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: s/librados2/librados3/

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: bump librados so version

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: bump version number

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: update test_envlibrados_for_rocksdb.sh for libradospp split

include a patch so rocksdb can use libradospp instead of librados. will
upstream the patch and make it work for both pre-nautilus librados and
nautilus libradospp

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: add libradospp* packages

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: add libradospp* packages

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: move C++ APIs into libradospp

the goal is to decouple C++ API from C API, and to version them
differently, as they are targeting different consumers.

this allows us to change the C++ API and bumping up its soversion
without requiring consumer to recompile the librados client for
using the new librados. in this way, C++ API can move faster than
C API. for example, if bufferlist interface is changed for better
performance, and this breaks existing API/ABI, we can bump up
the C++ library's soversion, and and the C library's version unchanged
but ship the new librados's C binding. so the librados client linked
against librados's C library will be able to take advantage of
the improvement in C++ library. while the librados client
linked against C++ library won't break at runtime due to unresolved
symbol or changed structure layout.

this is massive change, the genereal idea is to

* split librados.cc into two source files: librados_c.cc and
  librados_cxx.cc, the former for implementing C APIs, the later
  for C++ APIs.
* extract the C++ API in librados into librados-cxx, the library
  name will be libradospp. but we can change it before nautilus
  is released.
* link these librados libraries with static libraries which it
  depends on, so "-Wl,--exclude-libs,ALL" link flags can help
  hide the non-public symbols.
* extract the tests exercising librados' C++ API into a different
  source file named *_cxx.cc. for instance, to move the C++ tests
  in aio.cc into aio_cxx.cc
* extract the shared helper functions which do not use any librados
  or librados-cxx APIs into test_shared{.cc,h}. the "shared" here
  means, *shared* by C++ and C tests.
* extract the test fixtures, i.e., the subclasses of testing::Test,
  for testing C++ APIs into testcase_cxx.cc.
* update qa/workunits/rados/test.sh accordingly to add the splitted
  tests
* update the consumers of librados to link against librados-cxx
  instead, if they are using the C++ API.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: move get_inconsistent_pgs() into RadosClient

so we don't need parse the pg string and render it again for librados C
API.

the downside of this change is that, get_inconsistent_pgs() could be
implemented using RadosClient, instead be *in* it. but before we have
a place for the helper functions for these higher-level functions,
RadosClient is a good place for hosting them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24406 from ivancich/wip-rgw-renew-reshard-lock

rgw: renew resharding locks to prevent expiration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20598 from rishabh-d-dave/fix-rbdfuse-look-for-ceph-conf

rbd-fuse: look for ceph.conf in standard locations

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24829 from p-na/pna-fix-docstring

mgr/dashboard: Fix links to external documentation

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

doc/rados/operations/crush-map-edits: fix 'take' syntax

Fixes: https://tracker.ceph.com/issues/21496
Signed-off-by: Remy Zandwijk <remy@luckyhands.nl>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21767 from cfanz/wip-fix-rgw-race-condition

rgw: avoid race condition in RGWHTTPClient::wait()

Merge pull request #24810 from rhcs-dashboard/35691-landing-page-chart-improvements

mgr/dashboard: Landing Page: chart improvements

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #20794 from ghost/wip-nfs-doc-err

doc: replace rgw_namespace_expire_secs with rgw_nfs_namespace_expire_secs

Merge pull request #21599 from cfanz/wip-fix-rgw-build-error

rgw: fix radosgw-admin build error

Merge pull request #19983 from gohighsec/validate_hitset

mon:validate hit_set values before set

Merge pull request #24848 from joke-lee/wip-doc-radosgw-admin

rgw: add --op-mask in radosgw-admin help info

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24840 from ceph/wip-rm36648

ceph-volume systemd import main so console_scripts work for executable

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

rgw: add --op-mask in radosgw-admin help info

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge PR #19411 into master

* refs/pull/19411/head:
osd: Step down RecoveryCtx from class to struct

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #19288 into master

* refs/pull/19288/head:
bluestore/BlueRocksEnv: Object initialization after deletion

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge PR #19142 into master

* refs/pull/19142/head:
doc/dev/kernel-client-troubleshooting: Add kernel dynamic debuggin

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge PR #19146 into master

* refs/pull/19146/head:
doc/dev/logs: Removes non-existence file path

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #19726 into master

* refs/pull/19726/head:
osd: Sanity check, if too full or not

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #24809 into master

* refs/pull/24809/head:
os/bluestore: omit redundant '/' in OSD path for ceph-bluestore-tool if
os/bluestore: improve error handling for migrate ops in
qa/standtalone/osd-bluefs-volume-ops: remove redundant code.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #21009 into master

* refs/pull/21009/head:
rados: fix error message for cache(-try)-flush-evict-all commands

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #24534 into master

* refs/pull/24534/head:
librados: Copy buffer on AIO read

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24545 into master

* refs/pull/24545/head:
os/bluestore: debug_omit_block_device_write isn't always respected.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24548 into master

* refs/pull/24548/head:
Objecter: add ignore cache flag if got redirect reply

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge PR #24599 into master

* refs/pull/24599/head:
os/bluestore/KernelDevice: Honor discard_granularity

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24742 into master

* refs/pull/24742/head:
osd: slightly refactor PrimaryLogPG::do_op.
osd: avoid unnecessary map lookups in PrimaryLogPG::do_op.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

ceph-volume systemd import main so console_scripts work for executable

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22132 from yehudasa/wip-menv

ceph-menv: mrun shell environment

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #24755 into master

* refs/pull/24755/head:
mgr/diskprediction: use global device_failure_prediction_mode setting

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24764 from tspmelo/wip-jest-ci

mgr/dashboard: Reduce Jest logs in CI

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #24633 from Devp00l/wip-issue-36467

mgr/dashboard: Add a unit test form helper class

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Cleanup imports

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Fix links to documentations

- Added anchor to Grafana in URL of OSD, Overall Performance
- Fixed URL in Object Gateway, Buckets

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge PR #24668 into master

* refs/pull/24668/head:
mgr/DaemonState: fix get_config_defaults locking
mgr: adjust DaemonState device index when daemon metadata is updated
mgr/DaemonState: clear devices in set_metadata

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #24773 from sileht/no-scripts

ceph-volume: use console_scripts

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

rbd-fuse: look for ceph.conf in standard locations

Fixes: http://tracker.ceph.com/issues/12219
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Renamed CdFormGroups inital test description

Fixes: https://tracker.ceph.com/issues/36467
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use form helper in dashboard tests

Using form helper in pool, RGW user, user and role form tests and
although in the cd-validator tests.

Fixes: https://tracker.ceph.com/issues/36467
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #24715 from ricardoasmarques/fix-36564

mgr/dashboard: Fix ts error on iSCSI page

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Add a unit test form helper class

This class helps testing forms, most methods were outsourced from the
pool form test suite.

Fixes: https://tracker.ceph.com/issues/36467
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #24735 from b-ranto/wip-rpm-dashboards

build/ops: rpm: Package grafana dashboards

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23569 from jcsp/wip-kubejacker

script/kubejacker: include cls libs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22833 from rjfd/wip-dashboard-jwt

mgr/dashboard: JWT authentication

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Landing Page: chart improvements
- Updated chart colors.
- Added border for chart slices (visual impairment help).
- Adjusted chart vertical centering.

Fixes: https://tracker.ceph.com/issues/35691
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #21243 from majianpeng/osd-misc-cleanup

osd/ReplicatedBackend: remove useless assert

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd/ReplicatedBackend: remove useless assert.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #13925 from nvedant07/bug_fix_1

osdmaptool/cleanup: Completed osdmaptool's usage

Merge pull request #24791 from falcon78921/wip-docs-demodocs

doc: added demo document changes section

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

os/bluestore: omit redundant '/' in OSD path for ceph-bluestore-tool if
needed.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: run-frontend-unittests: fix CEPH_ROOT initialization

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: frontend: JWT authentication implementation

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: run-backend-api-request.sh: support for JTW tokens

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: backend: JWT based authentication

Signed-off-by: Ricardo Dias <rdias@suse.com>

os/bluestore: improve error handling for migrate ops in
ceph_bluestore-tool.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: Fix ts error on iSCSI page

This error only happens until initiator is connected to the target.

Fixes: https://tracker.ceph.com/issues/36564
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge PR #24686 into master

* refs/pull/24686/head:
os/bluestore: show compress and buffered from WriteContext
os/bluestore: fix rename race with trim on replacement onode at old name

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

mgr/diskprediction: use global device_failure_prediction_mode setting

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24229 into master

* refs/pull/24229/head:
common: drop BL_BACKWARD_COMPAT in bufferlist.
osd: OSDMap encoding uses bufferlist::contiguous_filler.
mds: encode_xattrs() uses buffer::list::contiguous_filler.
common: introduce contiguous_filler to optimize ENCODE_START.
common: optimize hole appending in bufferlist.
common: duplicate an encoding macro to suppress warnings.
common: drop backward iteration from bufferlist.

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #24787 into master

* refs/pull/24787/head:
Merge PR #24796 into nautilus
osd: fix heartbeat_reset unlock
Merge PR #24780 into nautilus
Merge PR #24761 into nautilus
Merge PR #24651 into nautilus
osd: fix race between op_wq and context_queue
test: Make sure kill_daemons failure will be easy to find
test: Add flush_pg_stats to make test more deterministic

qa/standtalone/osd-bluefs-volume-ops: remove redundant code.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #24763 from zmc/wip-36416

mgr/dashboard: Map dev 'releases' to master

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #24796 into nautilus

* refs/pull/24796/head:
osd: fix heartbeat_reset unlock

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: fix heartbeat_reset unlock

Fixes 51d8e2457d73c709bfa4f706793696b3ce704ff9, which moved to lock_guard
but didn't remove the unlock call on this exit path.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24774 into master

* refs/pull/24774/head:
cmake: update dpdk drivers/modules to accomodate dpdk submodule
spdk: update to latest v18.07

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #24666 into master

* refs/pull/24666/head:
include/types: fixed compile warning for signed/unsigned comparison
osd/PrimaryLogPG: uncommitted dup ops should respond with logged return code
osd/PrimaryLogPG: propagate error return codes on object copy_get ops
osd/PGLog: optionally record error return codes for extra_reqids
osd/osd_types: include PG log return codes in object copy data

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24688 into master

* refs/pull/24688/head:
common: make ceph_abort store same crash info as ceph_assert
global: store assert msg in global and dump to crash meta
pybind/mgr: make 'ceph crash ls' output sorted list
log: don't clear ring when dump_recent is called
ceph-crash: make clear to user that 'posted' should be directory

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24780 into nautilus

* refs/pull/24780/head:
osd: take heartbeat_lock before checking for session

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge PR #24780 into master

* refs/pull/24780/head:
osd: take heartbeat_lock before checking for session
Merge PR #24725 into nautilus
qa/tasks/qemu: use unique clone directory to avoid race with workunit
mds: add missing mds_lock

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge PR #24778 into master

* refs/pull/24778/head:
rpm: use %license macro for packaging license file

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #24792 from falcon78921/wip-doc-grammar1

doc: fixed typo in man page

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24702 from wjwithagen/wjw-fix-blkdev-serial-const

common: mark BlkDev::serial() const to match with its declaration

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fixed minor grammar error

Changed buchket to bucket

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

doc: added demo document changes section

Added a brief section about how to demo Ceph documentation
changes.

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

osd: slightly refactor PrimaryLogPG::do_op.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #24785 from falcon78921/wip-docs-36605

doc: purge subcommand link broken

Reviewed-by: Kefu Chai <kchai@redhat.com>