mon,mgr: track available modules in MgrMap too

Require --force to enable a module that isn't reported as available.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: update docs for enabling mgr modules

Signed-off-by: Sage Weil <sage@redhat.com>

mon,mgr: manage mgr module list in mgrmap

Kill old mgr_modules option.

Add new mgr_initial_modules option, on the mon, for the initial cluster
mgrmap.

Add ls, enable, disable commands.

Respawn mgr if the module list changes.  In the future we could enable
new modules without a full restart, but disabling probably requires (and
is best handled by) a respawn.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: include MgrMap in ClusterState

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15927 from jcsp/wip-mgr-log-on-load-failed

mgr: cluster log message on plugin load error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15951 from wido/issue-20430

osd: Execute crush_location_hook when configured in ceph.conf

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15954 from Liuchang0812/wip-osd-compact

mon: fix kvstore type in mon compact command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15957 from liewegas/wip-rc

ceph_release: we are in the 'rc' phase (12.1.z)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15968 from xiexingguo/wip-poolopts-fix

mon/OSDMonitor: two pool opts related fix

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15974 from liewegas/wip-require-osd-release-print

osd/OSDMap: print require_osd_release

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15976 from liewegas/wip-bluestore-big-cache

os/bluestore: default cache size of 3gb

Merge pull request #15489 from xxhdx1985126/master

librados: redirect balanced reads to acting primary when targeting object isn't recovered

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14678 from joscollin/wip-cleanup-storeget-return-check

mon: handle cases where store->get() may return error

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15926 from liewegas/wip-log-channel

librados: add log channel to rados_monitor_log2 callback

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15925 from jcsp/wip-fsstatus-cleanup

mgr: clean up fsstatus module

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15932 from dillaman/wip-20421

pybind: restore original API for backwards compatibility

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14760 from dzafman/wip-19657

osd: On EIO from read recover the primary replica from another copy

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15912 from cloudbau/wip-20107

rgw: Fix duplicate tag removal during GC

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Tested-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #15894 from liewegas/wip-no-default-pool

mon,osd: do not create rbd pool by default

Merge pull request #15471 from tchaikov/wip-radostriper-leak-on-fail

libradosstriper: fix MultiAioCompletion leaks on failure

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test: Fix osd-scrub-repair for movement of shards

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Clean up extra json output from log

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Give time for OSDs to see map with flag change

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #15812 from dzafman/wip-cot-cleanup

tools: Cleanup dead code in ceph-objectstore-tool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15983 from cbodley/wip-20386

rgw: fix infinite loop in rest api for log list

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15635 from dzafman/wip-13837

osd: Corrupt objects stop snaptrim and mark pg snaptrim_error

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15987 from liewegas/wip-19994

doc/release-notes: fix ceph-deploy command

doc/release-notes: fix ceph-deploy command

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: rest handlers for mdlog and datalog list dont loop

the do-while loops are unnecessary, because cls_log_list() will already
give us the requested number of entries, unless truncated. and these
rest operations return the truncated flag and last_marker, which the
client side is already using to run the same loop

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix RGWMetadataLog::list_entries() for null last_marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWMetadataLog::list_entries() no longer splices

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15850 from rzarzynski/wip-rgw-20384

rgw: fix wrong error code for expired Swift TempURL's links.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15883 from rzarzynski/wip-rgw-20395

rgw: omit X-Account-Access-Control if there is no grant to serialize.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15884 from joscollin/wip-1412617-uninitialized-pointer-field

rgw: initialize non-static class members in ESQueryCompiler

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15886 from joscollin/wip-CID1412616-uninitialized-pointer-field

rgw: initialize Non-static class member worker in RGWReshard

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15888 from joscollin/wip-1412615-uninitialized-scalar-field

rgw: initialize Non-static class member val in ESQueryNodeLeafVal_Int

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15913 from smithfarm/wip-20417

tests: rgw.py: put client roles in a separate list

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test,qa/workunits: fix a zillion tests

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: do not start mapping job on empty map

The Mapper doesn't do the completion.  Just avoid it in the caller instead
of adding a Finisher or something similarly annoying for the completion.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: create rbd pool after starting the osds

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/ceph-helpers: shorten mon connect timeout

Signed-off-by: Sage Weil <sage@redhat.com>

osdmaptool: do not create default pool without --with-default-pool

Signed-off-by: Sage Weil <sage@redhat.com>

test: update TestPGLog's pool_id for the new world (no pool 0)

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osdmap: maintain a pool-creating build_simple() for use in unit tests

Change the standard build_simple() so it no longer takes pg[p]_num -- these
params were unused. Add a new build_simple_with_pool() that can be called
when we want the rbd pool to be created by default. These are implemented
via a simple switched function (marked private).

Update the existing callers so they choose the right one for their needs.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osdmap: reference the correct value in debug output

Looks like we've been printing out the (possibly-still-default-initialized)
num_osd member variable instead of the given nosd value! Luckily
I don't see any other misuse and we fill out

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

cephtool/test: create an rbd pool since it's no longer automatic

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

TestOSDMap: provide more flexibility in the PrimaryAffinity check

Fix the margins for the primary affinity test.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>

osdmap: do not automatically create an rbd pool on cluster creation; skip pool 0

We don't use pool ID 0 any more because CephFS can't handle it.
Update the user quick-start to include it, and add some links to support that process.
Fix the tests to handle the new state of life.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

TestOSDMap: actually loop through the pools as intended

Looks like this was meant to be a loop but (unlike the other tests)
just referred to a constant pool 0. Whoops!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

pgmap: extend mon_pg_warn_min_objects to cover the PG-per-OSD warnings too

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

src/jobs: remove these definitely-broken old testing scripts

They haven't been touched in 6 years and were considered broken then.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

os/bluestore: default cache size of 3gb

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: print require_osd_release

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15967 from Adirl/continous

msg/async/rdma: register buffer as continuous

Reviewed-by: Haomai Wang <haomai@xsky.com>

fix infinite loop in rest api for log list

as max_entries_str is empty, it will be infinite loop when log is more than MAX_ENTRIES(1000) from marker.
and max_entries maybe overflow if use it to count the left entries.

Fixes: http://tracker.ceph.com/issues/20386
Signed-off-by: xierui <jerry.xr86@gmail.com>

Merge pull request #14102 from renhwztetecs/renhw-wip-moninit-cleanup

mon: clean up in ceph_mon.cc

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Jos Collin <jcollin@redhat.com>

os/bluestore: allow user to unset pool compression_mode[algorithm] properly

A string-typed pool option requires user pass in a empty string to do a valid cancellation,
but the "osd pool set " command won't allow it. E.g.:
./bin/ceph osd pool set rbd compression_mode
Invalid command:  missing required parameter val(<string>)

Since we already use the "unset" keyword to cancel the csum_type setting,
we could simply extends the above mechanism for compression_mode
and compression_algorithm too.

E.g.:
./bin/ceph osd pool set rbd compression_algorithm zlib
set pool 0 compression_algorithm to zlib

 ./bin/ceph osd pool set rbd compression_algorithm unset
unset pool 0 compression_algorithm

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

msg/async/rdma: register buffer as continuous

Change-Id: I1a715de28ace3ff8ae077ca783ad5060e7966d54
Signed-off-by: Adir Lev <adirl@mellanox.com>

mon: fix kvstore type in mon compact command

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

mon/OSDMonitor: fix error handling if we fail to set some pool opts

Returning a positive error code is generally not the preferred option
and can not be handled properly by caller here.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15907 from tchaikov/wip-with-zfs

cmake: fix the build with -DWITH_ZFS=ON

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15908 from Adirl/twice

msg/async/rdma: use lists properly

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15521 from dillaman/wip-rbd-dashboard

pybind/mgr/dashboard: initial block integration

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15853 from liewegas/wip-simpler-ceph

qa/tasks/ceph: simplify ceph deployment slightly

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph: explicitly add osds to crush map for upgrades

Before kraken, ceph-osd didn't add itself to crush... ceph-osd-prestart.sh
did it.  And ceph.py doesn't use that.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_release: we are in the 'rc' phase (12.1.z)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15955 from liewegas/wip-20431

qa/tasks/dump_stuck: fix for active+clean+remapped

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/dump_stuck: fix for active+clean+remapped

In d24a8886658c2d8882275d69c6409717a62701be we made remapped a clean
state but didn't fix this test.

Fixes: http://tracker.ceph.com/issues/20431
Signed-off-by: Sage Weil <sage@redhat.com>

cmake: compile libzfs backend conditionally

* do not REQUIRE libzfs if it is enabled, this follows the way how we
  handle `WITH_XFS` option.
* also refactor the cmake script related to libxfs backend support a
  little bit.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15874 from elliswuuuuuuuu/remove_unuse_config

common/config_opt: remove unused config

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15864 from joscollin/wip-CID1412839-uninitialized-pointer-field

common: initialize array in struct BackTrace

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15881 from joscollin/wip-1412776-explicit-null-dereferenced

common: Passing null pointer option_name to operator << in md_config_t::parse_option()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15896 from Yan-waller/wip-waller-mgrcleanup0624

mgr: drop repeated log info. and unnecessary write permission

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15910 from wido/cest-api-fork

ceph-rest-api: Various REST API fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15915 from liupan1111/wip-add-perfs

src/OSD: add more useful perf counters for performance tuning.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15917 from xiexingguo/wip-upmap-fixes

mon/OSDMonitor: a couple of upmap and other fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15935 from zphj1987/master

ceph-conf: fix typo in usage: 'mon add' should be 'mon addr'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15885 from liewegas/wip-radosbench-timeout

qa/tasks/radosbench: increase timeout

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #15882 from rzarzynski/wip-20372

rgw: fixes for AWSBrowserUploadAbstractor auth

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15931 from cbodley/wip-rgw-slo-compression

rgw: use uncompressed size for range_to_ofs() in slo/dlo

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15928 from liewegas/wip-rgw-warning

rgw/rgw_op: fix whitespace and indentation warning

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

mon: handle cases where store->get() may return error

Handled cases where store->get() may return error. Fixed the review comments too in this commit.
Fixes: http://tracker.ceph.com/issues/19601
Signed-off-by: Jos Collin <jcollin@redhat.com>

osd: Execute crush_location_hook when configured in ceph.conf

CrushLocation only invoked update_from_conf() which meant that
update_from_hook() was never executed.

By called init_on_startup() both functions are executed so that
the hook is called when configured.

Signed-off-by: Wido den Hollander <wido@42on.com>

qa: call into `status` module from cephtool test

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15920 from wido/docs-mgr-dashboard

docs: mgr dashboard

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <jspray@redhat.com>

msg/async/rdma: avoid adding worker twice to pending_workers

Change-Id: I4dda9040a055bdb8787ffd23a6928925d0a69d83
Signed-off-by: Adir Lev <adirl@mellanox.com>

msg/async/rdma: avoid adding csi twice to pending_list

Change-Id: Id76ffbfebc54c2da339e0e7f063c5420042fbe7c
Signed-off-by: Adir lev <adirl@mellanox.com>

doc: Mgr Dashboard docs update

More instructions on how to deploy the dashboard using ceph-mgr

Signed-off-by: Wido den Hollander <wido@42on.com>

tests: rgw.py: put client roles in a separate list

Fixes: http://tracker.ceph.com/issues/20417
Signed-off-by: Nathan Cutler <ncutler@suse.com>

common: enable `status` module by default

It's still sort of awkward to prefix these commands
with "mgr tell" but this makes them at least
somewhat accessible to the average user.

Signed-off-by: John Spray <john.spray@redhat.com>

pybind/mgr: rename fsstatus->status

Still not exactly the best name but
reflects that it isn't just the fs bit.

Signed-off-by: John Spray <john.spray@redhat.com>

pybind/mgr: clean up 'fs status' command

Handle the `fs` argument.

Signed-off-by: John Spray <john.spray@redhat.com>

pybind/mgr: clean up osd bit of fsstatus.py

Actually implement the bucket filtering argument,
rename the command to "osd status" to deconflict
with existing "osd perf" command.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: clean up in ceph_mon.cc

The output prints to the ceph-mon.log,
so need replace cout with dout,in addition, keep cout in the usage().

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #15919 from dillaman/wip-test-cls-rbd

test: fix compile warning in ceph_test_cls_rbd

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

mgr: cluster log message on plugin load error

To make it a bit more obvious what's going on, otherwise
user's first sign is when they try and use a CLI bit
and get a command not found.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15934 from myoungwon/omw-mailmap

mailmap: add Myoungwon Oh's mailmap and affiliation

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #15914 from wjwithagen/remotes/origin/wip-wjw-freebsd-osd-scrub-repair-3

test/osd/osd-scrub-repair.sh: Fix diff options on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15856 from cbodley/wip-rgw-vstart-s3tests

vstart: remove rgw_enable_static_website

Merge pull request #15863 from gmayyyha/paxosservice-use-func

mon/PaxosService: use __func__ instead of hard code function name

Reviewed-by: Kefu Chai <kchai@redhat.com>