Merge pull request #15585 from dachary/wip-20229-ceph-disk-timeout

ceph-disk: set the default systemd unit timeout to 3h

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15984 from idryomov/wip-chooseargs-encoding-fix

crush/CrushWrapper: chooseargs encoding fix

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16023 from liewegas/wip-19994

systemd/ceph-mgr: remove automagic mgr creation hack

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15956 from dillaman/wip-librbd-devstack

test: fix failing rbd devstack teuthology test

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15923 from dillaman/wip-qa-rbd-memcheck-python

qa/suites/rbd: restrict python memcheck validation to CentOS

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15911 from xiexingguo/wip-bluestore-20170626

os/bluestore: narrow cache lock range; make sure min_alloc_size p2 aligned

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #16044 from liewegas/wip-i-cant-believe-its-not-btrfs

qa: stop testing btrfs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16046 from liewegas/wip-valgrind-on-centos

do all valgrind runs on centos

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: recommend against btrfs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15963 from Songweibin/wip-release-notes

doc/release-notes: add Images creation timestamp note

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15924 from jcsp/wip-log-last-filters

mon: Filter `log last` output by severity and channel

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15946 from Yan-waller/wip-waller-dumpwatcher0627

osd: dump the field name of object watchers  and cleanups

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15959 from smithfarm/wip-rgw-singleton-cleanup

tests: rgw/singleton: drop duplicate filestore-xfs.yaml

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15961 from majianpeng/filestore-remove-redundant-judgement

os/filestore/FileStore.cc: remove a redundant judgement when get max latency

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

Merge pull request #15999 from Yan-waller/wip-waller-0629message

msg/MOSDOpReply: fix missing trace decode

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16037 from gmayyyha/monmapmonitor-use-func

mon/MonmapMonitor: use __func__ instead of hard code function name

Reviewed By: Joao Eduardo Luis <jecluis@gmail.com>

do all valgrind runs on centos

We are fighting two issues with valgrind on ubuntu (xenial, yakkety,
and z):

http://tracker.ceph.com/issues/18126
http://tracker.ceph.com/issues/20360

Revert this when it is fixed.

Signed-off-by: Sage Weil <sage@redhat.com>

qa: stop testing btrfs

This is too noisy and we don't recommend it anyway.

See http://tracker.ceph.com/issues/20169

Signed-off-by: Sage Weil <sage@redhat.com>

PendingReleaseNotes: add a note on choose_args encoding

It's unlikely that it's been used yet, but worth a note.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

crush/CrushWrapper: crush_choose_arg::ids should be __s32

crush_choose_arg::ids array is encoded on the wire.  int is not fixed
size -- use __s32 instead (crush_bucket::id is __s32).

This was introduced in commit dbe36e08be00 ("crush: compile/decompile
crush_choose_arg_map") under SERVER_LUMINOUS bit.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

crush/CrushWrapper: encode choose_args.size() as __u32

choose_args.size() returns size_t, which is not fixed size.  Given that
choose_args code just open-codes our std::map encoder, do what we do for
std::map.

This was introduced in commit dbe36e08be00 ("crush: compile/decompile
crush_choose_arg_map") under SERVER_LUMINOUS bit.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mon/MonmapMonitor: use __func__ instead of hard code function name

Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>

Merge pull request #15927 from jcsp/wip-mgr-log-on-load-failed

mgr: cluster log message on plugin load error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15951 from wido/issue-20430

osd: Execute crush_location_hook when configured in ceph.conf

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15954 from Liuchang0812/wip-osd-compact

mon: fix kvstore type in mon compact command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15957 from liewegas/wip-rc

ceph_release: we are in the 'rc' phase (12.1.z)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15968 from xiexingguo/wip-poolopts-fix

mon/OSDMonitor: two pool opts related fix

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15974 from liewegas/wip-require-osd-release-print

osd/OSDMap: print require_osd_release

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15976 from liewegas/wip-bluestore-big-cache

os/bluestore: default cache size of 3gb

Merge pull request #15489 from xxhdx1985126/master

librados: redirect balanced reads to acting primary when targeting object isn't recovered

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14678 from joscollin/wip-cleanup-storeget-return-check

mon: handle cases where store->get() may return error

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15926 from liewegas/wip-log-channel

librados: add log channel to rados_monitor_log2 callback

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15925 from jcsp/wip-fsstatus-cleanup

mgr: clean up fsstatus module

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15932 from dillaman/wip-20421

pybind: restore original API for backwards compatibility

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14760 from dzafman/wip-19657

osd: On EIO from read recover the primary replica from another copy

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15912 from cloudbau/wip-20107

rgw: Fix duplicate tag removal during GC

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Tested-by: Yuri Weinstein <yweinste@redhat.com>

systemd/ceph-mgr: remove automagic mgr creation hack

For kraken we auto-created mgr daemons next to mon daemons with some
systemd hackery.  This is awkward (you can't not get a new mgr daemon when
you deploy a mon), systemd-specific (not implemented for upstart on
trusty), and mostly unexpected.  Since ceph-mgr daemons are now first-class
citizens and required for every cluster, make their deployment explicit
and transparent to the administrator.  Major upgrades are a rare
opportunity to have the administrator's full attention so take advantage
of it.

This effectively reverts 61d779345e9efbe9a2e3f215af1f1dcf6630f04a and
082199f69dd0bd4c18a5f4baea67a88782586657 (and follow-on fixes).

Fixes/avoids: http://tracker.ceph.com/issues/19994
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15894 from liewegas/wip-no-default-pool

mon,osd: do not create rbd pool by default

mon: make `log` command send messages to default channel

...rather than leaving channel blank.  This means that
people watching the main "cluster" channel will see
the log messages sent using `log`.

Signed-off-by: John Spray <john.spray@redhat.com>

message: fix missing decode trace

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #15471 from tchaikov/wip-radostriper-leak-on-fail

libradosstriper: fix MultiAioCompletion leaks on failure

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test: Fix osd-scrub-repair for movement of shards

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Clean up extra json output from log

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Give time for OSDs to see map with flag change

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #15812 from dzafman/wip-cot-cleanup

tools: Cleanup dead code in ceph-objectstore-tool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15983 from cbodley/wip-20386

rgw: fix infinite loop in rest api for log list

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15635 from dzafman/wip-13837

osd: Corrupt objects stop snaptrim and mark pg snaptrim_error

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15987 from liewegas/wip-19994

doc/release-notes: fix ceph-deploy command

doc/release-notes: fix ceph-deploy command

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: rest handlers for mdlog and datalog list dont loop

the do-while loops are unnecessary, because cls_log_list() will already
give us the requested number of entries, unless truncated. and these
rest operations return the truncated flag and last_marker, which the
client side is already using to run the same loop

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix RGWMetadataLog::list_entries() for null last_marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWMetadataLog::list_entries() no longer splices

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15850 from rzarzynski/wip-rgw-20384

rgw: fix wrong error code for expired Swift TempURL's links.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15883 from rzarzynski/wip-rgw-20395

rgw: omit X-Account-Access-Control if there is no grant to serialize.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15884 from joscollin/wip-1412617-uninitialized-pointer-field

rgw: initialize non-static class members in ESQueryCompiler

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15886 from joscollin/wip-CID1412616-uninitialized-pointer-field

rgw: initialize Non-static class member worker in RGWReshard

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15888 from joscollin/wip-1412615-uninitialized-scalar-field

rgw: initialize Non-static class member val in ESQueryNodeLeafVal_Int

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15913 from smithfarm/wip-20417

tests: rgw.py: put client roles in a separate list

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test,qa/workunits: fix a zillion tests

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: do not start mapping job on empty map

The Mapper doesn't do the completion.  Just avoid it in the caller instead
of adding a Finisher or something similarly annoying for the completion.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: create rbd pool after starting the osds

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/ceph-helpers: shorten mon connect timeout

Signed-off-by: Sage Weil <sage@redhat.com>

osdmaptool: do not create default pool without --with-default-pool

Signed-off-by: Sage Weil <sage@redhat.com>

test: update TestPGLog's pool_id for the new world (no pool 0)

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osdmap: maintain a pool-creating build_simple() for use in unit tests

Change the standard build_simple() so it no longer takes pg[p]_num -- these
params were unused. Add a new build_simple_with_pool() that can be called
when we want the rbd pool to be created by default. These are implemented
via a simple switched function (marked private).

Update the existing callers so they choose the right one for their needs.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osdmap: reference the correct value in debug output

Looks like we've been printing out the (possibly-still-default-initialized)
num_osd member variable instead of the given nosd value! Luckily
I don't see any other misuse and we fill out

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

cephtool/test: create an rbd pool since it's no longer automatic

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

TestOSDMap: provide more flexibility in the PrimaryAffinity check

Fix the margins for the primary affinity test.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>

osdmap: do not automatically create an rbd pool on cluster creation; skip pool 0

We don't use pool ID 0 any more because CephFS can't handle it.
Update the user quick-start to include it, and add some links to support that process.
Fix the tests to handle the new state of life.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

TestOSDMap: actually loop through the pools as intended

Looks like this was meant to be a loop but (unlike the other tests)
just referred to a constant pool 0. Whoops!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

pgmap: extend mon_pg_warn_min_objects to cover the PG-per-OSD warnings too

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

src/jobs: remove these definitely-broken old testing scripts

They haven't been touched in 6 years and were considered broken then.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

os/bluestore: default cache size of 3gb

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: print require_osd_release

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15967 from Adirl/continous

msg/async/rdma: register buffer as continuous

Reviewed-by: Haomai Wang <haomai@xsky.com>

fix infinite loop in rest api for log list

as max_entries_str is empty, it will be infinite loop when log is more than MAX_ENTRIES(1000) from marker.
and max_entries maybe overflow if use it to count the left entries.

Fixes: http://tracker.ceph.com/issues/20386
Signed-off-by: xierui <jerry.xr86@gmail.com>

Merge pull request #14102 from renhwztetecs/renhw-wip-moninit-cleanup

mon: clean up in ceph_mon.cc

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Jos Collin <jcollin@redhat.com>

os/bluestore: allow user to unset pool compression_mode[algorithm] properly

A string-typed pool option requires user pass in a empty string to do a valid cancellation,
but the "osd pool set " command won't allow it. E.g.:
./bin/ceph osd pool set rbd compression_mode
Invalid command:  missing required parameter val(<string>)

Since we already use the "unset" keyword to cancel the csum_type setting,
we could simply extends the above mechanism for compression_mode
and compression_algorithm too.

E.g.:
./bin/ceph osd pool set rbd compression_algorithm zlib
set pool 0 compression_algorithm to zlib

 ./bin/ceph osd pool set rbd compression_algorithm unset
unset pool 0 compression_algorithm

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/filestore/FileStore.cc: remove a redundant judgement when get max latency.

In fact, utime_t default is utime_t(). So we can remove this judgement.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

msg/async/rdma: register buffer as continuous

Change-Id: I1a715de28ace3ff8ae077ca783ad5060e7966d54
Signed-off-by: Adir Lev <adirl@mellanox.com>

os/bluestore: fix writing buffers may get pinned

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon: fix kvstore type in mon compact command

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

mon/OSDMonitor: fix error handling if we fail to set some pool opts

Returning a positive error code is generally not the preferred option
and can not be handled properly by caller here.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc/release-notes: add Images creation timestamp note
Signed-off-by: songweibin song.weibin@zte.com.cn

Merge pull request #15907 from tchaikov/wip-with-zfs

cmake: fix the build with -DWITH_ZFS=ON

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15908 from Adirl/twice

msg/async/rdma: use lists properly

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15521 from dillaman/wip-rbd-dashboard

pybind/mgr/dashboard: initial block integration

Reviewed-by: John Spray <john.spray@redhat.com>

tests: rgw/singleton: drop duplicate filestore-xfs.yaml

This yaml is now included in the objectstore/ directory. It makes no sense to
duplicate it here.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #15853 from liewegas/wip-simpler-ceph

qa/tasks/ceph: simplify ceph deployment slightly

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph: explicitly add osds to crush map for upgrades

Before kraken, ceph-osd didn't add itself to crush... ceph-osd-prestart.sh
did it.  And ceph.py doesn't use that.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_release: we are in the 'rc' phase (12.1.z)

Signed-off-by: Sage Weil <sage@redhat.com>

qa/task/qemu: ensure test has access to all generated log files

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: preserve devstack configuration files

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: install noVNC from packaging

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rbd: sanity check cluster connection before devstack runs

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rbd: ensure guest VM updates to Ceph version under test

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15955 from liewegas/wip-20431

qa/tasks/dump_stuck: fix for active+clean+remapped

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/dump_stuck: fix for active+clean+remapped

In d24a8886658c2d8882275d69c6409717a62701be we made remapped a clean
state but didn't fix this test.

Fixes: http://tracker.ceph.com/issues/20431
Signed-off-by: Sage Weil <sage@redhat.com>

cmake: compile libzfs backend conditionally

* do not REQUIRE libzfs if it is enabled, this follows the way how we
  handle `WITH_XFS` option.
* also refactor the cmake script related to libxfs backend support a
  little bit.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15874 from elliswuuuuuuuu/remove_unuse_config

common/config_opt: remove unused config

Reviewed-by: Kefu Chai <kchai@redhat.com>