script: add docker core dump debugger

This script sets up a docker container with all the packages used for your Ceph
(test) branch. From there, you can gdb the core dump if you have access to the
test cluster.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15587 from wjwithagen/wip-wjw-ceph-disk-is_diskdevice

ceph-disk/ceph_disk/main.py: Replace ST_ISBLK() test by is_diskdevice()

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16347 from tchaikov/wip-test-ceph-disk

tests: ceph-disk: use communicate() instead of wait() for output

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #16334 from wjwithagen/wjw-bug-stringyfy

core:" Stringify needs access to << before reference" src/include/stringify.h

Reviewed-by: Jos Collin <jcollin@redhat.com>

qa/suites/ceph-disk: whitelist health warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

tests: ceph-disk: use communicate() instead of wait() for output

to avoid possible deadlock. quote from doc of Popen.wait()

> This will deadlock when using stdout=PIPE and/or stderr=PIPE and the
child process generates enough output to a pipe such that it blocks
waiting for the OS pipe buffer to accept more data. Use communicate() to
avoid that.

and print out the stdout and stderr using LOG.warn() if the command
fails.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16045 from Liuchang0812/wip-compact-osd-feature

osd: compact osd feature

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16221 from liewegas/wip-20546

crush/CrushWrapper: make get_immediate_parent[_id] ignore per-class shadow hierarchy

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #12139 from tianshan/fix_zone_set_realm_id

rgw: fix zone did't update realm_id when added to zonegroup

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #14915 from cbodley/wip-19817

rgw: add missing RGWPeriod::reflect() based on new atomic update_latest_epoch()

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15953 from rzarzynski/wip-rgw-20418

rgw: reject request if decoded URI contains \0 in the middle.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #16340 from dillaman/wip-20630

qa/tasks: rbd-mirror daemon not properly run in foreground mode

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16331 from zhangsw/cleanup-zonegroup-list

rgw: remove the useless output when listing zonegroups.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/tasks: rbd-mirror daemon not properly run in foreground mode

Fixes: http://tracker.ceph.com/issues/20630
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16043 from jcsp/wip-dashboard-updates

mgr: dashboard improvements

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: update health display

This takes account of the new health format, also
expands and visually cleans up the frontpage
where we put the health information.

Dark backgrounds make it much easier to use
red/amber/green colours to grab attention.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: expose a MgrMap in PyModules

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #16020 from jcsp/wip-20383

mgr: clean up daemon start process

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16338 from scienceluo/wip-doc-branch

doc/release-notes: Luminous release notes typo fixes

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc/release-notes: Luminous release notes typo fixes "systemctl ceph-osd.target"->"systemctl restart ceph-osd.target" and "systemctl ceph-mgr.target"->"systemctl restart ceph-mgr.target"

Signed-off-by: Luo Kexue <luo.kexue@zte.com.cn>

Merge pull request #16318 from smithfarm/wip-jewel-10-2-9

doc: Jewel v10.2.9 release notes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph_disk/main.py: fixed bugs from theutology report

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

ceph-disk/main.py: Replace ST_ISBLK() test by is_diskdevice()

 - FreeBSD does not have blockdevices any more (since 2002)
   So disk are just Character special devices, so test on ISCHR

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

core:" Stringify needs access to << before reference" src/include/stringify.h

Clang complains:

In file included from /home/jenkins/workspace/ceph-master/src/mon/HealthMonitor.cc:21:
/home/jenkins/workspace/ceph-master/src/include/stringify.h:15:6: error: call to function 'operator<<' that is neither visible in the template definition nor found by argument-dependent lookup
  ss << a;
     ^
/home/jenkins/workspace/ceph-master/src/mon/HealthMonitor.cc:129:32: note: in instantiation of function template specialization 'stringify<std::__1::set<std::__1::basic_string<char>, std::__1::less<std::__1::basic_string<char> >, std::__1::allocator<std::__1::basic_string<char> > > >' requested here
      boost::regex("%names%"), stringify(names[p.first]));
                               ^
/home/jenkins/workspace/ceph-master/src/include/types.h:160:17: note: 'operator<<' should be declared prior to the call site
inline ostream& operator<<(ostream& out, const set<A, Comp, Alloc>& iset) {

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #16330 from scienceluo/master

doc/release-notes: Luminous release notes typo fixes  "ceph config-key ls"->"ceph config-key list"

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc:fix release-notes

Signed-off-by: luo.kexue@zte.com.cn

Merge pull request #16300 from liewegas/wip-20600

mon: fix hang on deprecated/removed 'pg set_*full_ratio' commands

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16321 from Yan-waller/wip-walle-0712cephosd

common: misc cleanups in common, global, os, osd submodules

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16322 from liewegas/wip-20617

qa/tasks/ceph_manager: wait longer for pg stats to flush

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16323 from ceph/revert-15897-wip-20390

Revert "msg/async: increase worker reference with local listen table enabled backend"

Merge pull request #16319 from tchaikov/wip-ceph-helper-with-exp-features

qa/workunits/ceph-helpers: enable experimental features for osd

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #16320 from tchaikov/wip-clang-analyzer-warnings

test,mon,msg: kill clang analyzer warnings

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

rgw: remove the useless output when listing zonegroups.

Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

doc: Jewel v10.2.9 changelog

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mon/PGMonitor: EOPNOTSUPP for old pgmon commands

This includes 'pg set_full_ratio', which we have only for the upgrade, but
goes away afterwards.

Also, return true to either swallow the request or indicate it has been
processed.

Fixes: http://tracker.ceph.com/issues/20600
Signed-off-by: Sage Weil <sage@redhat.com>

mon: do not assign to never-read variable

this silences clang analyzer's warning of

Value stored to 'err' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async/rdma: return stored errno on error

otherwise the errno would be overwritten, and we are returning 0 or the
errno set by ::close()

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: test_c_headers: silence clang analyzer warnings

this silences clang analyzer's warnings like:

Value stored to 'ret' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16262 from liewegas/wip-20208

mgr/ClusterState: do not mangle PGMap outside of Incremental

Merge pull request #9974 from weiqiaomiao/wqm-wip-copy_obj

rgw: fix memory leak in copy_obj_to_remote_dest

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11124 from zhangsw/cleanup-rgwrados-deleteobj

rgw: remove a redundant judgement in rgw_rados.cc:delete_obj.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12010 from zhangsw/fix-rgw-multipart-bug

rgw: Fix a bug that multipart upload may exceed the quota.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

osdmaptool: add --health option

Generate and dump health checks.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test_health_warnings: test down msgs vs crush classes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12197 from zhangsw/fix-rgw-metasync-lock-bug

rgw: lock is not released when set sync marker is failed.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd/OSDMap: remove assumption about type ids

The code is assuming type==1 is in use, but it might not be.  (It is
usually 'chassis' by default, which is rarely used; 'host' is type usually
type 2.)  Remove the type check entirely and identify leaves by a child
>= 0.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: make get_immediate_parent[_id] ignore per-class shadow hierarchy

Fixes: http://tracker.ceph.com/issues/20546
Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: wait longer for pg stats to flush

An ill-timed mgr restart could blow the current 15s wait.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16021 from joscollin/wip-uninitialized-pointer-fields-1

rgw: Initialize pointer fields

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #16014 from xiexingguo/wip-autoclass

osd/OSD: auto class on osd start up

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15774 from shashalu/drop-temp-var

rgw: using RGW_OBJ_NS_MULTIPART in check_bad_index_multipart

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: Jewel v10.2.9 release notes

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Revert "msg/async: increase worker reference with local listen table enabled backend"

Merge pull request #16255 from trociny/wip-test-librbd-internals

test/librbd: re-enable internal tests in ceph_test_librbd

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16130 from fullerdj/wip-djf-ceph-connect-timeout

ceph.in: Check return value when connecting

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #16243 from markhpc/wip-bluestore-freelist-iterator

os/bluestore: Make BitmapFreelistManager kv itereator short lived.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16269 from liewegas/wip-bluestore-deferred-pending

os/bluestore: only submit deferred if there is any

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #16306 from liewegas/wip-reg11184-health

qa/suites/rados/singleton/all/reg11184: whitelist health warnings

osd: cleanups

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

test: add test of ceph compact command

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

osd: new command compact via tell/daemon

user could manual compact OSD's omap as following:
1. ceph tell osd.id compact
2. ceph daemon osd.id compact
user's requests will be impacted during compaction.

Fixes: http://tracker.ceph.com/issues/19592
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

os: export compact interface in ObjectStore and ObjectMap

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #16317 from tchaikov/wip-0-osd-is-not-an-error

qa/workunits/ceph-helpers: test wait_for_health_ok differently

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #16274 from smithfarm/wip-jewel-10-2-8

doc: Jewel v10.2.8 release notes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #16267 from liewegas/wip-restful-defaults

mgr/restful: bind to :: and update docs

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #16311 from Songweibin/wip-update-info-affi

.mailmap, .organizationmap: Update Song Weibin information and affiliation

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #16303 from bstillwell/releases-doc-update-201707

doc/releases: Update releases from Feb 2017 to July 2017

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: add v10.2.8 changelog

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/workunits/ceph-helpers: enable experimental features for osd

it matches the settings in vstart.sh, also it would be handy for those
who are still developing on btrfs, which is now marked as an experimental
features now.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/ceph-helpers: test wait_for_health_ok differently

0 OSDs is not an error anymore in the new health checking implemented by
OSDMap::check_health(). this case was treated as an error before, see
OSDMonitor::get_health(). but an osdmap without any OSD is fine, i
think. but an osdmap with 3 OSDs, but all of them are down and out, this
is an error. and we do report this as an error. so, let's update the
test instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16277 from smithfarm/wip-rn-script

tools: ceph-release-notes: handle an edge case

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: release notes: note MDS regression in 10.2.8

See the discussion in https://github.com/ceph/ceph/pull/16192

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #16264 from dillaman/wip-20571

rbd-mirror: ignore permission errors on rbd_mirroring object

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

.mailmap, .organizationmap: Update Song Weibin information and affiliation

Signed-off-by: songweibin <song.weibin@zte.com.cn>

qa/suites/rados/singleton/all/reg11184: whitelist health warnings

Signed-off-by: Sage Weil <sage@redhat.com>

Update releases from Feb 2017 to July 2017

Pull in the latest releases from the past 5 months and fix some of the
links so they jump to the correct sections in the release notes.

Signed-off-by: Bryan Stillwell <bstillwell@godaddy.com>

Merge pull request #16301 from dmick/master

mgr: increase debug level for ticks 0 -> 10

Mgr: increase debug level for ticks 0 -> 10

Signed-off-by: Dan Mick <dmick@redhat.com>

mon/MonCommands: mark 'pg set_*_ratio' deprecated

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16263 from liupan1111/wip-fix-fio

test/fio: remove experimental option for bluestore & rocksdb.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15643 from liewegas/wip-health

mon: revamp health check/warning system

mon/PGMap: adjust scrub checks to avoid overflow for future stamps

Avoid an overflow (and false warning) when scrub stamps are in the future.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: adjust full tests to avoid races

OSDs may report fullness in any order.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: wait for osds to come up before creating pool

Avoid health warnings.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_test_case.py: update health check helpers

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/fs: whitelist health warnings

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rgw/thrash: whitelist

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rbd: whitelist health messages

Signed-off-by: Sage Weil <sage@redhat.com>

qa: whitelist health warnings

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: adjust for new health error codes

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: clear last_beacon after mon election

The last_beacon map is local to an election interval; if there is a new
election completed we should reset it or else we may kill an apparently
laggy mgr that hasn't been able to get a beacon processed due to the mon
quorum changing, or had its beacon processed on a different leader.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: clean up `osd out` messages

Cleaner prose for the auto-out case, and add
a cluster log message for OSDs that go out
at the behest of the administrator.

Signed-off-by: John Spray <john.spray@redhat.com>

osd: don't log per-PG backfill messages at INFO level

This behaviour led to way too many messages going to
the cluster log when an OSD is marked in.  Retain
the messages at debug level.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: simplify PG health checks

Instead of a distinct health check for each possible
PG state, group the states into categories for availability,
degraded, damage, and report on that.

That way, while a PG/pool is suffering from one of those
bad PG states, health conditions don't keep toggling on and
off as we transition from one unavailable state to another
unavailable state.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: update for new style health checks

Signed-off-by: John Spray <john.spray@redhat.com>

mon: demote cluster map prints to DEBUG level

The PaxosService subclasses should be writing out
informative log messages, and not relying on
a stream of map summary prints to communicate
changes.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: prettify health check log messages

Add a "Cluster is now healthy" to give clarity
after a series of "health check cleared" that
they were the last ones.

Convert certain health check messages into
well formed sentences.

Don't print severity in the log string (it's already
expressed in the severity of the log entry.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fix spurious PG health messages on mgr restart

Previously, the mgr would send MMonMgrReport indicating
a very unhappy PGMap to the mon right after startup.

This is a change to hold off on sending that report until
all the OSDs have reported in, or until some time has passed.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: don't consider a starting mgr to be an error

The .available flag is there to tell MgrClients whether
to try and connect -- it isn't the right condition
for a health complaint.

Signed-off-by: John Spray <john.spray@redhat.com>