systemd/ceph-mgr: remove automagic mgr creation hack

For kraken we auto-created mgr daemons next to mon daemons with some
systemd hackery.  This is awkward (you can't not get a new mgr daemon when
you deploy a mon), systemd-specific (not implemented for upstart on
trusty), and mostly unexpected.  Since ceph-mgr daemons are now first-class
citizens and required for every cluster, make their deployment explicit
and transparent to the administrator.  Major upgrades are a rare
opportunity to have the administrator's full attention so take advantage
of it.

This effectively reverts 61d779345e9efbe9a2e3f215af1f1dcf6630f04a and
082199f69dd0bd4c18a5f4baea67a88782586657 (and follow-on fixes).

Fixes/avoids: http://tracker.ceph.com/issues/19994
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15894 from liewegas/wip-no-default-pool

mon,osd: do not create rbd pool by default

Merge pull request #15471 from tchaikov/wip-radostriper-leak-on-fail

libradosstriper: fix MultiAioCompletion leaks on failure

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test: Fix osd-scrub-repair for movement of shards

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Clean up extra json output from log

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Give time for OSDs to see map with flag change

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #15812 from dzafman/wip-cot-cleanup

tools: Cleanup dead code in ceph-objectstore-tool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15983 from cbodley/wip-20386

rgw: fix infinite loop in rest api for log list

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15635 from dzafman/wip-13837

osd: Corrupt objects stop snaptrim and mark pg snaptrim_error

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15987 from liewegas/wip-19994

doc/release-notes: fix ceph-deploy command

doc/release-notes: fix ceph-deploy command

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: rest handlers for mdlog and datalog list dont loop

the do-while loops are unnecessary, because cls_log_list() will already
give us the requested number of entries, unless truncated. and these
rest operations return the truncated flag and last_marker, which the
client side is already using to run the same loop

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix RGWMetadataLog::list_entries() for null last_marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWMetadataLog::list_entries() no longer splices

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15850 from rzarzynski/wip-rgw-20384

rgw: fix wrong error code for expired Swift TempURL's links.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15883 from rzarzynski/wip-rgw-20395

rgw: omit X-Account-Access-Control if there is no grant to serialize.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15884 from joscollin/wip-1412617-uninitialized-pointer-field

rgw: initialize non-static class members in ESQueryCompiler

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15886 from joscollin/wip-CID1412616-uninitialized-pointer-field

rgw: initialize Non-static class member worker in RGWReshard

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15888 from joscollin/wip-1412615-uninitialized-scalar-field

rgw: initialize Non-static class member val in ESQueryNodeLeafVal_Int

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15913 from smithfarm/wip-20417

tests: rgw.py: put client roles in a separate list

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test,qa/workunits: fix a zillion tests

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: do not start mapping job on empty map

The Mapper doesn't do the completion.  Just avoid it in the caller instead
of adding a Finisher or something similarly annoying for the completion.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: create rbd pool after starting the osds

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/ceph-helpers: shorten mon connect timeout

Signed-off-by: Sage Weil <sage@redhat.com>

osdmaptool: do not create default pool without --with-default-pool

Signed-off-by: Sage Weil <sage@redhat.com>

test: update TestPGLog's pool_id for the new world (no pool 0)

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osdmap: maintain a pool-creating build_simple() for use in unit tests

Change the standard build_simple() so it no longer takes pg[p]_num -- these
params were unused. Add a new build_simple_with_pool() that can be called
when we want the rbd pool to be created by default. These are implemented
via a simple switched function (marked private).

Update the existing callers so they choose the right one for their needs.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osdmap: reference the correct value in debug output

Looks like we've been printing out the (possibly-still-default-initialized)
num_osd member variable instead of the given nosd value! Luckily
I don't see any other misuse and we fill out

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

cephtool/test: create an rbd pool since it's no longer automatic

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

TestOSDMap: provide more flexibility in the PrimaryAffinity check

Fix the margins for the primary affinity test.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>

osdmap: do not automatically create an rbd pool on cluster creation; skip pool 0

We don't use pool ID 0 any more because CephFS can't handle it.
Update the user quick-start to include it, and add some links to support that process.
Fix the tests to handle the new state of life.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

TestOSDMap: actually loop through the pools as intended

Looks like this was meant to be a loop but (unlike the other tests)
just referred to a constant pool 0. Whoops!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

pgmap: extend mon_pg_warn_min_objects to cover the PG-per-OSD warnings too

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

src/jobs: remove these definitely-broken old testing scripts

They haven't been touched in 6 years and were considered broken then.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15967 from Adirl/continous

msg/async/rdma: register buffer as continuous

Reviewed-by: Haomai Wang <haomai@xsky.com>

fix infinite loop in rest api for log list

as max_entries_str is empty, it will be infinite loop when log is more than MAX_ENTRIES(1000) from marker.
and max_entries maybe overflow if use it to count the left entries.

Fixes: http://tracker.ceph.com/issues/20386
Signed-off-by: xierui <jerry.xr86@gmail.com>

Merge pull request #14102 from renhwztetecs/renhw-wip-moninit-cleanup

mon: clean up in ceph_mon.cc

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Jos Collin <jcollin@redhat.com>

msg/async/rdma: register buffer as continuous

Change-Id: I1a715de28ace3ff8ae077ca783ad5060e7966d54
Signed-off-by: Adir Lev <adirl@mellanox.com>

Merge pull request #15907 from tchaikov/wip-with-zfs

cmake: fix the build with -DWITH_ZFS=ON

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15908 from Adirl/twice

msg/async/rdma: use lists properly

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15521 from dillaman/wip-rbd-dashboard

pybind/mgr/dashboard: initial block integration

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15853 from liewegas/wip-simpler-ceph

qa/tasks/ceph: simplify ceph deployment slightly

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph: explicitly add osds to crush map for upgrades

Before kraken, ceph-osd didn't add itself to crush... ceph-osd-prestart.sh
did it.  And ceph.py doesn't use that.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15955 from liewegas/wip-20431

qa/tasks/dump_stuck: fix for active+clean+remapped

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/dump_stuck: fix for active+clean+remapped

In d24a8886658c2d8882275d69c6409717a62701be we made remapped a clean
state but didn't fix this test.

Fixes: http://tracker.ceph.com/issues/20431
Signed-off-by: Sage Weil <sage@redhat.com>

cmake: compile libzfs backend conditionally

* do not REQUIRE libzfs if it is enabled, this follows the way how we
  handle `WITH_XFS` option.
* also refactor the cmake script related to libxfs backend support a
  little bit.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15874 from elliswuuuuuuuu/remove_unuse_config

common/config_opt: remove unused config

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15864 from joscollin/wip-CID1412839-uninitialized-pointer-field

common: initialize array in struct BackTrace

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15881 from joscollin/wip-1412776-explicit-null-dereferenced

common: Passing null pointer option_name to operator << in md_config_t::parse_option()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15896 from Yan-waller/wip-waller-mgrcleanup0624

mgr: drop repeated log info. and unnecessary write permission

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15910 from wido/cest-api-fork

ceph-rest-api: Various REST API fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15915 from liupan1111/wip-add-perfs

src/OSD: add more useful perf counters for performance tuning.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15917 from xiexingguo/wip-upmap-fixes

mon/OSDMonitor: a couple of upmap and other fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15935 from zphj1987/master

ceph-conf: fix typo in usage: 'mon add' should be 'mon addr'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15885 from liewegas/wip-radosbench-timeout

qa/tasks/radosbench: increase timeout

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #15882 from rzarzynski/wip-20372

rgw: fixes for AWSBrowserUploadAbstractor auth

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15931 from cbodley/wip-rgw-slo-compression

rgw: use uncompressed size for range_to_ofs() in slo/dlo

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15928 from liewegas/wip-rgw-warning

rgw/rgw_op: fix whitespace and indentation warning

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15920 from wido/docs-mgr-dashboard

docs: mgr dashboard

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <jspray@redhat.com>

msg/async/rdma: avoid adding worker twice to pending_workers

Change-Id: I4dda9040a055bdb8787ffd23a6928925d0a69d83
Signed-off-by: Adir Lev <adirl@mellanox.com>

msg/async/rdma: avoid adding csi twice to pending_list

Change-Id: Id76ffbfebc54c2da339e0e7f063c5420042fbe7c
Signed-off-by: Adir lev <adirl@mellanox.com>

doc: Mgr Dashboard docs update

More instructions on how to deploy the dashboard using ceph-mgr

Signed-off-by: Wido den Hollander <wido@42on.com>

tests: rgw.py: put client roles in a separate list

Fixes: http://tracker.ceph.com/issues/20417
Signed-off-by: Nathan Cutler <ncutler@suse.com>

mon: clean up in ceph_mon.cc

The output prints to the ceph-mon.log,
so need replace cout with dout,in addition, keep cout in the usage().

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #15919 from dillaman/wip-test-cls-rbd

test: fix compile warning in ceph_test_cls_rbd

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15934 from myoungwon/omw-mailmap

mailmap: add Myoungwon Oh's mailmap and affiliation

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #15914 from wjwithagen/remotes/origin/wip-wjw-freebsd-osd-scrub-repair-3

test/osd/osd-scrub-repair.sh: Fix diff options on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15856 from cbodley/wip-rgw-vstart-s3tests

vstart: remove rgw_enable_static_website

Merge pull request #15863 from gmayyyha/paxosservice-use-func

mon/PaxosService: use __func__ instead of hard code function name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15759 from ovh/bp-optimized-bs-key-escape

os/bluestore: optimized (encode|decode)_escaped

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15765 from jimmyway/wip-ec-plan-cleanup

osd/ECTransaction: cleanup the redundant check which works in overwrite IO context

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-conf: fix typo in usage: 'mon add' should be 'mon addr'

src/test/cli/ceph-conf/help.t
src/tools/ceph_conf.cc

fix add to addr

Signed-off-by: Peng Zhang zphj1987@gmail.com

mailmap: add Myoungwon Oh's mailmap and affiliation

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

rgw: use uncompressed size for range_to_ofs() in slo/dlo

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rgw_op: fix some more whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

rgw/rgw_op: fix indentation warning

/home/sage/src/ceph2/src/rgw/rgw_op.cc: In member function ‘virtual void RGWDeleteObjTags::execute()’:
/home/sage/src/ceph2/src/rgw/rgw_op.cc:777:3: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
   if (s->object.empty())
   ^~
/home/sage/src/ceph2/src/rgw/rgw_op.cc:780:5: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the ‘if’
     rgw_obj obj;
     ^~~~~~~

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/radosbench: increase timeout

The current timeout isn't enough in some cases (powercycle thrashing leaves
osds down for a long time because rebooting is so slow).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14997 from ivancich/wip-bring-in-dmclock-p2

osd/PG: Add two new mClock implementations of the PG sharded operator queue

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15569 from aclamk/compress_crash_bug_refactor

rgw: Compress crash bug refactor

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15859 from smithfarm/wip-swift-task-move-master

tests: move swift.py task from teuthology to ceph, phase one (master)

Merge pull request #15555 from liewegas/wip-19023

osd/PG: set clean when last_epoch_clean is updated

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15862 from yuyuyu101/wip-omap-enoent

kv/RocksDBStore: abort if rocksdb EIO, don't return incorrect result

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15866 from tchaikov/wip-set-secrets

auth/RotatingKeyRing: use std::move() to set secrets

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15905 from tchaikov/wip-kill-clang-warnings

test, ceph-osdomap-tool: kill clang warnings

Reviewed-by: Sage Weil <sage@redhat.com>

src/OSD: add more useful perf counters for performance tuning.

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

Merge pull request #13753 from theanalyst/wip/f/rgw/tagging

rgw: implement get/put object tags for S3

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15889 from joscollin/wip-1412614-uninitialized-scalar-field

tools/rbd_mirror: initialize non-static class member m_do_resync in ImageReplayer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test: fix compile warning in ceph_test_cls_rbd

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon/OSDMonitor: add use_gmt_hitset to "osd pool get" cmd filter

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: drop debug_fake_ec_pool from "osd pool set" cmd filter

It is dead!

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: fix wrong sanity check order for allow_ec_overwrites

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: make blacklist expire configurable; fix potential races

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: fix wrong comment, wrong indention, wrong output etc.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: strict comparison condition

To match the specifications of straw_calc_version.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/osd_type: map.find() -> map.count() for existense checking

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: kill dead code

This option is now filtered out automatically, so these codes are unaccessable.

./bin/ceph osd unset sortbitwise
Invalid command:  sortbitwise not in full|pause|noup|nodown|noout|noin|nobackfill|norebalance|norecover|noscrub|nodeep-scrub|notieragent
osd unset full|pause|noup|nodown|noout|noin|nobackfill|norebalance|norecover|noscrub|nodeep-scrub|notieragent :  unset <key>
Error EINVAL: invalid command

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: make pg-up-items result a bit nicer

~#: ./bin/ceph osd pg-upmap-items 0.7 0 2 1 3 4 5
set 0.7 pg_upmap_items mapping to [0->2,1->3,4->5]

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: reject updating upmap if number of osd overflows/underflows

This can avoid, e.g.:

~#:./bin/ceph osd pg-upmap 0.7 1 2 3 4
set 0.7 pg_upmap mapping to [1,2,3,4]

~#:./bin/ceph pg dump
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES LOG DISK_LOG STATE        STATE_STAMP                VERSION REPORTED UP        UP_PRIMARY ACTING    ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP                LAST_DEEP_SCRUB DEEP_SCRUB_STAMP
0.7           0                  0        0         0       0     0   0        0      peering 2017-06-24 10:11:03.938102     0'0    53:18 [1,2,3,4]          1 [1,2,3,4]              1        0'0 2017-06-24 09:38:50.162197             0'0 2017-06-24 09:38:50.162197

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMap: make apply_upmap handle multiple pg-upmap-items correctly

Was:
PG_STAT  UP      UP_PRIMARY ACTING  ACTING_PRIMARY
0.7      [1,0,4]          1 [1,0,4]              1

~#:./bin/ceph osd pg-upmap-items 0.7 0 2 1 3 4 5
set 0.7 pg_upmap_items mapping to [0,2,1,3,4,5]

PG_STAT  UP      UP_PRIMARY ACTING  ACTING_PRIMARY
0.7      [1,2,4]          1 [1,2,4]              1

Now:
PG_STAT  UP      UP_PRIMARY ACTING  ACTING_PRIMARY
0.7      [1,0,4]          1 [1,0,4]              1

~#:./bin/ceph osd pg-upmap-items 0.7 0 2 1 3 4 5
set 0.7 pg_upmap_items mapping to [0,2,1,3,4,5]

PG_STAT  UP      UP_PRIMARY ACTING  ACTING_PRIMARY
0.7      [3,2,5]          3 [3,2,5]              3

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/osd/osd-scrub-repair.sh: Fix diff options

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>