common/common_init: disable ms subsystem log gathering for clients

The log gathering causes large performance degradation to clients
with high message throughputs. This is hopefully a short-term
workaround until per-message logging can be replaced with an
efficient data recording system for post-incident analysis
use-cases.

Fixes: http://tracker.ceph.com/issues/21860
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18374 from tchaikov/wip-ceph-C-d

ceph.in: do not panic at control+d in interactive mode

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #17462 from majianpeng/pmem-work

cmake: compile nvml as an external project

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: Build libpmem from source code.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #18389 from yaozongyou/fix-type-in-bluestore-migration

doc: fix typo in bluestore-migration.rst

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18395 from gaosibei/mailmap

mailmap: Add Sibei, XueYu Affiliation

Reviewed-by: Jos Collin <jcollin@redhat.com>

mailmap, organizationmap: add Sibei, XueYu

Signed-off-by: Sibei Gao <gaosb@inspur.com>

doc: fix typo in bluestore-migration.rst

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #18232 from jdurgin/wip-filestore-rocksdb

common/options: enable multiple rocksdb compaction threads for filestore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18311 from shinobu-x/sk-osd_20171015

osd: Don't initialze pointers by NULL or 0

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #14213 from xiaoxichen/fix_wbthrottle_assert

os/Filestore: fix wbthrottle assert.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18256 from liewegas/wip-bluestore-superblock

os/bluestore: allow reconstruction of osd data dir from bluestore bdev label

Reviewed-By: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18346 from tchaikov/wip-log-if-do_read_entry-fails

os/filestore: print out the error if do_read_entry() fails

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18145 from dzafman/wip-18162

osd: object added to missing set for backfill, but is not in recovering, error!

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18377 from joscollin/wip-cleanup-rgw-function-names

rgw: Fix getter function names in RGWEnv

Merge pull request #18380 from zmc/wip-master-init

Changes required for teuthology's systemd support

Merge pull request #18365 from linuxbox2/wip-rgw-v3open-write

rgw_file: explicit NFSv3 open() emulation

qa/tasks/ceph: Set owner on mds mount point

Signed-off-by: Zack Cerza <zack@redhat.com>

qa/tasks/cephfs/filesystem: Check for mds failure

... inside Filesystem.are_daemons_healthy()

Signed-off-by: Zack Cerza <zack@redhat.com>

qa/tasks/ceph: Remove /var/lib/ceph copypasta

Signed-off-by: Zack Cerza <zack@redhat.com>

qa/tasks/ceph: Set owner on osd mount point

This is needed when running with systemd

Signed-off-by: Zack Cerza <zack@redhat.com>

qa/tasks/ceph: Set owner on mon mount point

This is needed when running with systemd

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #15554 from cbodley/wip-rgw-parse-cr-dump

tools/rgw: add script to inspect admin socket "cr dump"

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

test: Cleanup test-erasure-eio.sh code

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add replicated recovery/backfill test

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Better recovery/backfill restart for mark_unfound_lost delete/revert

Signed-off-by: David Zafman <dzafman@redhat.com>

osd, mon: Add new pg states recovery_unfound and backfill_unfound

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add new UnfoundBackfill and UnfoundRecovery pg transitions

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Test case that reproduces tracker 18162

recover_replicas: object added to missing set for backfill, but is not in recovering, error!

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #18378 from ceph/wip-yuriw-21822-master

tests - ceph-ansible vars additions

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>

tests - ceph-ansible vars additions
added symlinks for distros

Fixes http://tracker.ceph.com/issues/21822
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit f30e9a5e6bd47129f8a35c1f9067216cfe8a6a70)
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rgw: Fix getter function names in RGWEnv

Fixed the getter function names as per the review comment in
https://github.com/ceph/ceph/pull/17432

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18376 from tchaikov/wip-authors

doc: update Component Technical Leads and maintainers to canonical location

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18355 from gaosibei/wip-rgw-refrence

RGW: change function parameters from value to refrence

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17432 from joscollin/wip-cleanup-fix-rgwconf

rgw: Access RGWConf through RGWEnv

Merge pull request #17440 from mikulely/cleanup-iterate

rgw: update outdated debug func name

Merge pull request #17956 from zhangsw/fix-rgw-encry-torrent-bug

rgw: Torrents are not supported for objects encrypted using SSE-C

Merge pull request #18057 from baixueyu/branch1

rgw: fix radosgw-admin quota enable return value bug

Merge pull request #18375 from tchaikov/wip-ceph-disk-check-log

ceph-disk: write log to /var/log/ceph not to /var/run/ceph

Reviewed-by: Alfredo Deza <adeza@redhat.com>

AUTHORS: Update Component Technical Leads and maintainers to canonical location of ceph.com.

Signed-off-by: Patrick McGarry <pmcgarry@redhat.com>

Merge pull request #18315 from dillaman/wip-21797

librbd: object map batch update might cause OSD suicide timeout

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18345 from wido/doc-osd-replace

doc: Remove additional arguments when replacing OSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-disk: write log to /var/log/ceph not to /var/run/ceph

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.in: do not panic at control+d in interactive mode

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18373 from tchaikov/wip-opt-desc

doc: fix the desc of "osd max pg per osd hard ratio"

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: fix the desc of "osd max pg per osd hard ratio"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18370 from mikulely/cleanup-rgw-write-meta

rgw: drop outdated function doc

Reviewed-by: Jos Collin <jcollin@redhat.com>

rgw: drop misplaced&outdated func doc

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

Merge pull request #18352 from rzarzynski/wip-testfio-fix-pgps

Revert "bluestore/fio: Fixed problem with all objects having the same hash

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #18338 from xiexingguo/wip-pg

osd/PrimaryLogPG: do not use approx_size() for log trimming

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18340 from Yan-waller/wip-walle-1014osdprimarypg

osd/PG: misc cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18351 from tchaikov/wip-osx

common/event_socket.h: include <errno.h> to use errno

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18353 from xiexingguo/wip-do-read

osd/PrimaryLogPG: trigger auto-repair on full-object-size CRC error

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18321 from majianpeng/msg-async-cleanup

msg/async: batch handle numevents.

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18358 from tchaikov/wip-max-pg-per-osd

osd,mon: add max-pg-per-osd limit

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18289 from smithfarm/wip-21762

build/ops: rpm: move ceph-*-tool binaries out of ceph-test subpackage

Reviewed-By: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

RGW: change function parameters from value to refrence

Signed-off-by: Sibei Gao <gaosb@inspur.com>

rgw_file: explicit NFSv3 open() emulation

Expect RGW_OPEN_FLAG_V3 in rgw_write(), and if present, attempt
a stateless open inline.

Required by Ganesha v2.5 and later

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #18184 from oritwas/wip-rgw-21725

rgw: disable dynamic resharding in multisite environment

Merge pull request #16745 from Liuchang0812/object-store-repair

common, tool: update kvstore-tool to repair key/value database

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #18281 into master

* refs/pull/18281/head:
Fixed the readme examples
ptl-tool: Drop Default Labelling of PRs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mon: change "mon_pg_warn_min_per_osd" to uint64_t

* this silences a warning from -Wsign-compare.
* also switch all its user to the new-style option

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd,mon: change "mon_max_pg_per_osd" to uint64_t

as it should never be a negative number. if this option is disabled,
it's 0.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: add max-pg-per-osd limit

osd will refused to create new pgs, until its pg number is lower
than the max-pg-per-osd upper bound setting.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18331 from Yan-waller/wip-walle-1008vstartrgw

vstart.sh: move rgw configuration to client.rgw section

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Fixed the readme examples

Fixed the readme examples, according to the new --label definition.

Signed-off-by: Jos Collin <jcollin@redhat.com>

ptl-tool: Drop Default Labelling of PRs

Dropped default labelling of PRs. Allowed --label '', which won't label the PR. The user can omit --label too, if the PR should not be labelled.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18231 from batrick/ptl-tool-labeled-prs

ptl-tool: load labeled PRs

Reviewed-by: Jos Collin <jcollin@redhat.com>

osd/PrimaryLogPG: trigger auto-repair on full-object-size CRC error

There are still FileStore users and the full-object CRC might be able
to catch silently returned data errors. Since we support auto-repair now,
let's trigger an auto-repair process instead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

common/lockdep: do not set std::string using braces

this silences follow warning from clang:

ceph/src/common/lockdep.cc:219:12: warning: braces around scalar initializer [-Wbraced-scalar-init]
    name = { "unknown" };
           ^~~~~~~~~~~~~
1 warning generated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "bluestore/fio: Fixed problem with all objects having the same hash"

This reverts commit 0eb5359273df2c92fa525683f88586878f57bb4b.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common/event_socket.h: include <errno.h> to use errno

OSX needs this also.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18348 from Yan-waller/wip-walle-1017readmecmake

README.md: bump up cmake to 2.8.12

Reviewed-by: Kefu Chai <kchai@redhat.com>

README.md: bump up cmake to 2.8.12

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

rgw: Access RGWConf through RGWEnv

Created functions rgw_conf_get_enable_ops_log(), rgw_conf_get_enable_usage_log() and
rgw_conf_get_defer_to_bucket_acls() and the necessary changes to provide the access of RGWConf through RGWEnv.

Signed-off-by: Jos Collin <jcollin@redhat.com>

os/filestore: print out the error if do_read_entry() fails

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: Remove additional arguments when replacing OSD

It's not required to provide the --bluestore nor --osd-uuid option
to ceph-disk.

Using BlueStore is the default on master (e.g. Luminous) and thus
this option is not required.

When no OSD UUID is supplied ceph-disk will generate one, so we do
not need to supply one generated by uuidgen.

Signed-off-by: Wido den Hollander <wido@42on.com>

osd/PrimaryLogPG: do not use approx_size() for log trimming

There might be holes on log versions, thus the approx_size()
should (almost) always overestimate the actual number of log entries.
As a result, we might be at the risk of accessing violation
while searching for the oldest log entry to keep in the log list later.

Fix the above problem by counting the precise number of current
log entries instead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

common/config: drop unused configuration item 'osd_backoff_on_down'

drop it for no more use.

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

rgw: update outdated debug func name

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

Merge pull request #18081 from zhsj/revert-15089-add-mirror

doc: change cn mirror to ustc domain

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18312 from shinobu-x/sk-buffer_20171015

common/buffer: cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18318 from Linbing/wip-bluestore-compression

bluestore:  comp_min_blob_size init error

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18279 from lixiaoy1/rmrange

os/bluestore: use db->rm_range_keys to delete range of keys

Reviewed-by: Varada Kari <varada.kari@sandisk.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: use db->rm_range_keys to delete range of keys

As Key Value DB provides the rm_range_keys interface to remove
some range of keys, BlueStore should use the interface. DB can
provide more efficient ways.

Signed-off-by: Xiaoyan Li <xiaoyan.li@intel.com>

Merge pull request #18320 from shinobu-x/sk-include_unordered_x

include: Remove unused header, ciso646

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #18319 from shinobu-x/sk-osd_20171016

osd: Don't include same header twice

Reviewed-by: Jos Collin <jcollin@redhat.com>

ptl-tool: load labeled PRs

This lets you label in GitHub the PRs you want to test/merge.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/ceph: add "create_rbd_pool" option

True by default, but we can opt not to create the "rbd" pool.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: bypass checks if pg_temp.forced

originally, monitor ignores the pg_temp messages sent from replica osds,
but to re-trigger peering if replica osd's pg number drops down below
the setting, we need to bypass the checks.

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages/MOSDPGTemp: add `forced` field

the monitor is supposed to bypass some checks if `forced` is true.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: print osd as "osd.$osdid" in log message

easier to read this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18332 from rzarzynski/wip-testfio-fix-build-error

test/fio: fix building of the fio_ceph_objectstore plugin.

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #17134 from PCzhangPC/metaexin

rbd: export/import image-meta when we export/import an image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18249 from PCzhangPC/addrwpro

rbd: show read:write proportion in the infomation of readwrite bench test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: fsync on _write_bdev_label

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-bluestore-tool: implement 'set-label-key' and 'rm-label-key'

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make _write_bdev_label static and public

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: batch handle numevents.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #18271 from cbodley/wip-21772

rgw: fix bilog entries on multipart complete

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>