mon: reduce CPU and memory manager pressure of pg health check

When we don't need details about which PGs are stuck, we can just iterate
once over pg_stat map and increase appropriate counters, no need to
repeatedly spam memory manager with stuck_pgs map population (we'll only
use its size). Even if we need details, first check if we actually have any
PGs in any of requested states, so in most cases we won't iterate needlessly
five times over all PG stat map.

This at least halves the time needed by pg health gathering (~3ms,
down from ~6-7ms) on cluster with 31832 PGs and Intel Xeon E5-2640
CPU.

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #7473 from yehudasa/wip-14516

rgw-admin: document orphans commands in usage

Merge pull request #7451 from wjin/fix

msg/async: fix array boundary

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6813 from manens/pybind_rados_snapshot_read

pybind: Implementation of Ioctx.set_read to allow read from snapshots

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw-admin: document orphans commands in usage

Fixes: #14516
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7445 from xiexingguo/xxg-wip-mem

osd: memstore: fix blocksize for statfs

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: fix typo (again)

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7400 from yuyuyu101/wip-13938

librbd: Avoid create two threads per image

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7409 from dillaman/wip-librbd-watch-notify

tests: improve simulation of watch/notify handling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6562 from Aran85/authedpermcheck

rgw: fix the condition of  authenticated user's permit check .

Merge pull request #7291 from majianpeng/vstart.sh

vstart.sh: If use '-n' delete related source rather than consider '-k'.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7397 from tchaikov/wip-kill-warnings

build: kill warnings

Reviewed-by: Sage Weil <sage@redhat.com>

civetweb: update submodule to silence a compile warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

common,osd: remove _process(T *t) to silence warnings

* and mark the WorkQueue::_process(T *, TPHandle&) a pure virtual
  method. so we must override it
* and mark all override methods with the "override" specifier.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: fix the condition of authenticated user's permit check

before fix:
put a bucket under one user with authenticated-read permit
> ./s3curl.pl http://myobs.com/mybucket --put --H "x-amz-acl: authenticated-read"
GET the bucket with anonymous user
> curl http://myobs.com/mybucket -X GET

> successed!

after fix:
anonymous cant GET the authenticated-read bucket
other users who authenticated could GET it

Signed-off-by: Zengran Zhang <zhangzengran@h3c.com>

Merge pull request #7266 from sahithi-rv/master

common/str_map: cleanup: replaced get_str_map() function overloading by using default parameters for delimiters

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7344 from songbaisen/song8

crush: add safety assert

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7363 from cxwshawn/mon-sync-fix

mon: add mon_config_key prefix when sync full

Fixes: #14577
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7383 from liewegas/wip-bluestore

osd: bluestore: some new c++11-isms, a couple reverts

Event: fix array boundary

fd should be less than nevent when deleting

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

cmake: fix "jobserver unavailable" warning

use ${MAKE} so the make command can be properly propagated to their
children

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: disable unused-variable warning for rocksdb

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: fix the signed/unsigned comparison warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

rados: bench: fix the signed/unsigned comparison warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove duplicated HAVE_EXECINFO_H macro

fixes the `"HAVE_EXECINFO_H" redefined` warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/*stores: fix hidden warnings

the CollectionHandle variant methods are hidden by the subclasses
who do not override it. so the compiler complains. by adding
"using" statement we expose them again.

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fix unsigned/signed comparision warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7014 from efirs/ef_replace_coll_t_by_const_coll_t_ref

osd: reduce CPU usage by replacing coll_t by const coll_t&

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7333 from xdonghai/xdh1

osd: FileStore: modify the format of colon

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7392 from tianshan/wip-14536

rados: fix race condition in load-gen complete callback

Reviewed-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #7395 from xiexingguo/xxg-wip-fuse

os/fusestore: add error handling

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7396 from tchaikov/wip-14537

mon: compact full epochs also

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7402 from liewegas/wip-rados-bench

rados: make rados bench metadata backward compatible

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7418 from renhwsky/renhw-wip-monmapmonitor-cleanup

mon: some cleanup in MonmapMonitor.cc

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #7420 from mikulely/fix-osd-mon-exit-issue

osd, mon: fix exit issue

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7430 from xiexingguo/xxg-wip-kstore

osd: kstore: fix decode logic and add flush

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7432 from branch-predictor/bp-fix-read-bench

rados: fix wrong op/object sizes in rand/seq bench

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7436 from liewegas/wip-bash

tests: qa/workunits/objectstore/test_fuse.sh: no bashism

os/memstore: return empty list if offset overflows

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/memstore: fix wrong blocksize for statfs

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #7404 from sahithi-rv/Mutex

rbd_fuse: replaced pthread_mutex*() functions with ceph Mutex wrapper

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tools/rados/rados.cc: fix race condition in load-gen complete callback

Fixes: #14536
Signed-off-by: Tianshan Qu <tianshan@xsky.com>

Merge pull request #6131 from rzarzynski/wip-admincreds-in-s3-keystone

rgw: support admin credentials in S3-related Keystone authentication.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: support admin credentials in S3-related Keystone authentication.

Fixes: #13302
Backport: infernalis, hammer
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #7298 from racpatel/wip-doc-rac2

doc :- fixing image in section ERASURE CODING

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph_osd.cc: fix unreachable flush call

The calling chain for generic_server_usage():

  generic_server_usage()
    ->generic_usage(true)
    ->exit(1)
  cout.flush()

any statements after generic_server_usage() would not be reached,so we
need to flush cout in generic_usage().

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

cmake: musl libc does not implement backtrace. Added feature check and fallback to libexecinfo.

Signed-off-by: John Coyle <dx9err@gmail.com>

Merge pull request #7264 from ceph/wip-doc-RGW

doc: rgw: port changes from downstream to upstream

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7244 from mikulely/fix-authtool-doc

doc: remove redundant space in ceph-authtool/monmaptool doc

doc: Updated the RPM section for RHEL and CentOS

Fixes: #10245
Signed-off-by: Aron Gunn <ritz_303@yahoo.com>

Merge pull request #7027 from mathslinux/fix-s3-csharp-doc

doc: fix S3 C# example

Merge pull request #7000 from zhouyuan/fix_cache_tier_doc

doc: fix misleading configuration guide on cache tiering

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6930 from wido/rgw-key-docs

doc: use 'ceph auth get-or-create' for creating RGW keyring

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7384 from wido/mirroring

doc: script and guidelines for mirroring Ceph

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #7435 from xiexingguo/xxg-wip-14561

osd: BlueStore: fix null pointer access

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7271 from somnathr/wip-transaction-delete

osd: avoid heap allocations for ObjectStore::Transaction

http://pulpito.ceph.com/sage-2016-01-28_12:50:51-rados-wip-sage-testing3---basic-mira/

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6440 from stiopaa1/scrubbingInfo

mon: warn if pg(s) not scrubbed

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #7075 from shun-s/shun-fix

global: do not start two daemons with a single pid-file

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7111 from chenji-kael/master

osd: FileStore: print file name before osd assert if read file failed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7268 from ifed01/wip-14400

common: snappy decompressor may assert when handling segmented input bufferlist

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7276 from caibo2014/fix-messager-null

mon/MonClient: avoid null pointer error when configured incorrectly

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7330 from ifed01/wip-pool-stat-cleanup

librados: remove duplicate definitions for rados pool_stat_t and cluster_stat_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7340 from ceph/wip-portable-time

common: ake ceph_time clocks work under BSD

Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge pull request #7341 from songbaisen/song7

mon: consider the pool size when setting pool crush rule

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7355 from yuanjingkai/master

os/fs: fix io_getevents argument

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7348 from xiexingguo/xxg-wip-14484

osd: BlueStore: fix space leak

http://pulpito.ceph.com/sage-2016-01-27_18:39:48-rados-wip-sage-testing---basic-smithi/

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7368 from somnathr/wip-jemalloc-build

build: allow jemalloc enabled build with rocksdb-static

http://pulpito.ceph.com/sage-2016-01-27_18:39:48-rados-wip-sage-testing---basic-smithi/

Reviewed-by: Sage Weil <sage@redhat.com>

BlueStore: fix null pointer access

Fixes: #14561
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/workunits/objectstore/test_fuse.sh: no bashism

Signed-off-by: Sage Weil <sage@redhat.com>

include/str_map.h removed unnecessary function overloading of
get_str_map()

Signed-off-by: Sahithi R V <sahithi.rv1@gmail.com>
tests successful

Signed-off-by: Sahithi R V <sahithi.rv1@gmail.com>

BlueStore: fix typo

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rbd_fuse:replaced pthread_mutex*() functions with ceph Mutex wrapper

Signed-off-by: Sahithi R V <tansy.rv@gmail.com>

kstore: flush before we really start a clone

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

kstore: flush before we really start a truncate

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

BlueStore: add fast check against empty list

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

kstore: add fast check against empty list

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

tools/rados: fix wrong op/object sizes in rand/seq bench

Metadata contain correct information about last bench write run, yet
these are ignored in subsequent bench rand/seq runs, resulting in
wrong perf numbers and failures when not using --no-verify.

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

kstore: simplify open_collection logic a bit

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

kstore: fix unmatched type of decode

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph_osd.cc/ceph_mon.cc: cleanup unreachable exit call

The calling chain of usage() is:

  usage()
    ->generic_server_usage()
      ->exit(1)
  exit(0)

so the exit(0) after usage() would not be reached.

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

mon:some cleanup in MonmapMonitor.h

delete unused "get_monmap(MonMap &m)"

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

mon:some cleanup in MonmapMonitor.cc

remove the processing of  ”mon add“ && “mon remove” in the preprocess_command;
delete unused "get_monmap(MonMap &m)"

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #7312 from jcsp/wip-14365

mds, client: fix locking around handle_conf_change

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge remote-tracking branch 'origin/greg-fs-testing-126'

Merge remote-tracking branch 'origin/jewel'

Merge pull request #7297 from ukernel/jewel-13903

client: properly trim unlinked inode

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7262 from ukernel/jewel-14380

Be more careful about the "mds setmap" command to prevent breakage.

Reviewed-by: John Spray <john.spray@redhat.com>

librbd: use task finisher per CephContext

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7369 from mauri/fix-message-typo

rbd: fix error message typo

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: watch/notify now behaves similar to librados

Notifications are executed via the same librados AIO callback
thread, so it's now possible to catch deadlock.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

FuseStore: fix mountpoint leak

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

FuseStore: return error if encounter collection_list error

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

FuseStore: fix unhandled error cases for open

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

FuseStore: fix memory leak

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #7387 from theanalyst/fix/gitignore

remove unused src/messages/#MOSDOp.h# file

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: simulate writeback flush during snap create

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7345 from x11562/rbd_snap_rename

rbd: add "--dest-snap" optional for snap rename

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7393 from tchaikov/wip-cmake

fix the autotools and cmake build (the new fusestore needs libfuse)

Reviewed-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #7403 from rzarzynski/wip-rgw-dumpconlen-cosmetics

rgw: cosmetic improvements in dump_content_length()