Merge pull request #18375 from tchaikov/wip-ceph-disk-check-log

ceph-disk: write log to /var/log/ceph not to /var/run/ceph

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18315 from dillaman/wip-21797

librbd: object map batch update might cause OSD suicide timeout

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18345 from wido/doc-osd-replace

doc: Remove additional arguments when replacing OSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-disk: write log to /var/log/ceph not to /var/run/ceph

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18373 from tchaikov/wip-opt-desc

doc: fix the desc of "osd max pg per osd hard ratio"

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: fix the desc of "osd max pg per osd hard ratio"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18370 from mikulely/cleanup-rgw-write-meta

rgw: drop outdated function doc

Reviewed-by: Jos Collin <jcollin@redhat.com>

rgw: drop misplaced&outdated func doc

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

Merge pull request #18352 from rzarzynski/wip-testfio-fix-pgps

Revert "bluestore/fio: Fixed problem with all objects having the same hash

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #18338 from xiexingguo/wip-pg

osd/PrimaryLogPG: do not use approx_size() for log trimming

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18340 from Yan-waller/wip-walle-1014osdprimarypg

osd/PG: misc cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18351 from tchaikov/wip-osx

common/event_socket.h: include <errno.h> to use errno

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18353 from xiexingguo/wip-do-read

osd/PrimaryLogPG: trigger auto-repair on full-object-size CRC error

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18321 from majianpeng/msg-async-cleanup

msg/async: batch handle numevents.

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18358 from tchaikov/wip-max-pg-per-osd

osd,mon: add max-pg-per-osd limit

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18289 from smithfarm/wip-21762

build/ops: rpm: move ceph-*-tool binaries out of ceph-test subpackage

Reviewed-By: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #18184 from oritwas/wip-rgw-21725

rgw: disable dynamic resharding in multisite environment

Merge pull request #16745 from Liuchang0812/object-store-repair

common, tool: update kvstore-tool to repair key/value database

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #18281 into master

* refs/pull/18281/head:
Fixed the readme examples
ptl-tool: Drop Default Labelling of PRs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mon: change "mon_pg_warn_min_per_osd" to uint64_t

* this silences a warning from -Wsign-compare.
* also switch all its user to the new-style option

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd,mon: change "mon_max_pg_per_osd" to uint64_t

as it should never be a negative number. if this option is disabled,
it's 0.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: add max-pg-per-osd limit

osd will refused to create new pgs, until its pg number is lower
than the max-pg-per-osd upper bound setting.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18331 from Yan-waller/wip-walle-1008vstartrgw

vstart.sh: move rgw configuration to client.rgw section

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Fixed the readme examples

Fixed the readme examples, according to the new --label definition.

Signed-off-by: Jos Collin <jcollin@redhat.com>

ptl-tool: Drop Default Labelling of PRs

Dropped default labelling of PRs. Allowed --label '', which won't label the PR. The user can omit --label too, if the PR should not be labelled.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18231 from batrick/ptl-tool-labeled-prs

ptl-tool: load labeled PRs

Reviewed-by: Jos Collin <jcollin@redhat.com>

osd/PrimaryLogPG: trigger auto-repair on full-object-size CRC error

There are still FileStore users and the full-object CRC might be able
to catch silently returned data errors. Since we support auto-repair now,
let's trigger an auto-repair process instead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

common/lockdep: do not set std::string using braces

this silences follow warning from clang:

ceph/src/common/lockdep.cc:219:12: warning: braces around scalar initializer [-Wbraced-scalar-init]
    name = { "unknown" };
           ^~~~~~~~~~~~~
1 warning generated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "bluestore/fio: Fixed problem with all objects having the same hash"

This reverts commit 0eb5359273df2c92fa525683f88586878f57bb4b.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common/event_socket.h: include <errno.h> to use errno

OSX needs this also.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18348 from Yan-waller/wip-walle-1017readmecmake

README.md: bump up cmake to 2.8.12

Reviewed-by: Kefu Chai <kchai@redhat.com>

README.md: bump up cmake to 2.8.12

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

doc: Remove additional arguments when replacing OSD

It's not required to provide the --bluestore nor --osd-uuid option
to ceph-disk.

Using BlueStore is the default on master (e.g. Luminous) and thus
this option is not required.

When no OSD UUID is supplied ceph-disk will generate one, so we do
not need to supply one generated by uuidgen.

Signed-off-by: Wido den Hollander <wido@42on.com>

osd/PrimaryLogPG: do not use approx_size() for log trimming

There might be holes on log versions, thus the approx_size()
should (almost) always overestimate the actual number of log entries.
As a result, we might be at the risk of accessing violation
while searching for the oldest log entry to keep in the log list later.

Fix the above problem by counting the precise number of current
log entries instead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

common/config: drop unused configuration item 'osd_backoff_on_down'

drop it for no more use.

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #18081 from zhsj/revert-15089-add-mirror

doc: change cn mirror to ustc domain

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18312 from shinobu-x/sk-buffer_20171015

common/buffer: cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18318 from Linbing/wip-bluestore-compression

bluestore:  comp_min_blob_size init error

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18279 from lixiaoy1/rmrange

os/bluestore: use db->rm_range_keys to delete range of keys

Reviewed-by: Varada Kari <varada.kari@sandisk.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: use db->rm_range_keys to delete range of keys

As Key Value DB provides the rm_range_keys interface to remove
some range of keys, BlueStore should use the interface. DB can
provide more efficient ways.

Signed-off-by: Xiaoyan Li <xiaoyan.li@intel.com>

Merge pull request #18320 from shinobu-x/sk-include_unordered_x

include: Remove unused header, ciso646

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #18319 from shinobu-x/sk-osd_20171016

osd: Don't include same header twice

Reviewed-by: Jos Collin <jcollin@redhat.com>

ptl-tool: load labeled PRs

This lets you label in GitHub the PRs you want to test/merge.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/ceph: add "create_rbd_pool" option

True by default, but we can opt not to create the "rbd" pool.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: bypass checks if pg_temp.forced

originally, monitor ignores the pg_temp messages sent from replica osds,
but to re-trigger peering if replica osd's pg number drops down below
the setting, we need to bypass the checks.

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages/MOSDPGTemp: add `forced` field

the monitor is supposed to bypass some checks if `forced` is true.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: print osd as "osd.$osdid" in log message

easier to read this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18332 from rzarzynski/wip-testfio-fix-build-error

test/fio: fix building of the fio_ceph_objectstore plugin.

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #17134 from PCzhangPC/metaexin

rbd: export/import image-meta when we export/import an image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18249 from PCzhangPC/addrwpro

rbd: show read:write proportion in the infomation of readwrite bench test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

msg/async: batch handle numevents.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #18271 from cbodley/wip-21772

rgw: fix bilog entries on multipart complete

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #18212 from shashalu/clean-up-radosgw-admin

rgw: drop "realm remove" in radosgw-admin

Reviewed-by: Jos Collin <jcollin@redhat.com>

test/fio: fix building of the fio_ceph_objectstore plugin.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #18313 from Carbenium/rbd-map-crash

rbd: fix crash during map when "rw" option is specified

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18238 from tchaikov/wip-bufferlist-test

test/unittest_bufferlist: check retvals of syscalls

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

vstart.sh: move rgw configuration to client.rgw section

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

build/ops: deb: move ceph-*-tool binaries out of ceph-test subpackage

ceph-osdomap-tool into ceph-osd subpackage
ceph-monstore-tool into ceph-mon subpackage
ceph-kvstore-tool into the ceph-base subpackage

Fixes: http://tracker.ceph.com/issues/21762
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #18129 from dragonylffly/wip-fix-coverity-warning-for-uninialized-member

librbd: fix coverity warning for uninitialized member

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18314 from MahatiC/example_librbd

examples: add new librbd example

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18265 from majianpeng/lirbd-cleanup

librbd: drop redundant check for null ImageCtx

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18326 from joscollin/wip-log-sign-compare-warning

log: silence warning from -Wsign-compare

Reviewed-by: Kefu Chai <kchai@redhat.com>

tool: do not core dump with a invalid kv type

Fixes: http://tracker.ceph.com/issues/21744
Signed-off-by: Chang Liu <liuchang0812@gmail.com>

test: new test case for ceph-kvstore-tool

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

doc: introduce repair subcommon of ceph-kvstore-tool

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

common, tool: update kvstore-tool to repair our key/value database

Fixes: http://tracker.ceph.com/issues/17730
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #18288 from trociny/wip-metadata

test/librbd: test metadata_set/remove is applied

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15941 from oritwas/wip-rgw-dynamic-resharding-doc

rgw: Add dynamic resharding documentation

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Alfredo Deza <adeza@redhat.com>

rbd:show read:write proportion in the infomation of readwrite bench test

Signed-off-by: PCzhangPC <pengcheng.zhang@easystack.cn>

Merge pull request #18209 from tchaikov/wip-new-style-opt-mon

mon/OSDMonitor: use new style options

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18093 from jcsp/wip-mgr-gil-safety

mgr: safety checks on pyThreadState usage

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

log: silence warning from -Wsign-compare

This warning appears in jenkins build:

/home/jenkins-build/build/workspace/ceph-pull-requests/src/log/test.cc:266:232:   required from here
/home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:1392:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (lhs == rhs) {

Signed-off-by: Jos Collin <jcollin@redhat.com>

mgr: safety checks on pyThreadState usage

Previously relied on the caller of Gil() to
pass new_thread=true if they would be
calling from a different thread.

Enforce this with an assertion, by wrapping
PyThreadState in a SafeThreadState class
that remembers which POSIX thread
it's meant to be used in.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: move Gil implementation into .cc

The inclusion of Python.h in the .h was awkward
for other files including Gil.h.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: reduce Gil verbosity at level 20

Even at 20, it's pretty heavy to be logging
every lock acquire/release.

Signed-off-by: John Spray <john.spray@redhat.com>

build/ops: rpm: move ceph-*-tool binaries out of ceph-test subpackage

ceph-osdomap-tool into ceph-osd subpackage
ceph-monstore-tool into ceph-mon subpackage
ceph-kvstore-tool into the ceph-base subpackage

Fixes: http://tracker.ceph.com/issues/21762
Signed-off-by: Nathan Cutler <ncutler@suse.com>

include/unordered_map/_set | cleanup: Remove unused header, ciso646

Signed-off-by: Shinobu Kinjo <shinobu@redhat.com>

osd | cleanup: Don't include same header twice

Signed-off-by: Shinobu Kinjo <shinobu@redhat.com>

bluestore: using bluestore_compression_min_blob_size instead of bluestore_compression_max_blob_size to config comp_min_blob_size

Signed-off-by: linbing <linbing@t2cloud.net>

Merge PR #18309 into master

* refs/pull/18309/head:
mds: prevent trim count from underflowing

Reviewed-by: Zheng Yan <zyan@redhat.com>

librbd: batch large object map updates into multiple chunks

Fixes: http://tracker.ceph.com/issues/21797
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: initial test cases for trim state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

examples: add new librbd example

Adds a librbd example

Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

librbd: tweaks to support testing of trim state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: combine trim state machine object map batch update states

The PRE/POST states were previously divided into two halves for handling
the copy-up batch and the direct removal batch. This can be simplified by
just using a single PRE/POST for the entire deletion region.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: object map update now utilizes constant-time bit vector operations

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common/bit_vector: provide constant time iteration of underlying bufferlist

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common/buffer: expose hidden const deference operator

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: fix crash during map

Currently the iterator isn't advanced after the erase call leading to a
second call on the iterator, which crashes due to a double free.
Since C++11 the map::erase function returns an iterator pointing to the
next element. Use the return value to set the iterator after erasing.

Fixes: http://tracker.ceph.com/issues/21808
Signed-off-by: Peter Keresztes Schmidt <carbenium@outlook.com>

rgw: Add dynamic resharding documentation

Fixes: http://tracker.ceph.com/issues/21553
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

osd/PrimaryLogPG: replace soid.get_head() with head

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

common | cleanup: buffer.cc

Signed-off-by: Shinobu Kinjo <shinobu@redhat.com>

Merge pull request #18292 from xiexingguo/wip-recovery-order

osd/PrimaryLogPG: arrange recovery order by number of missing objects

Reviewed-by: Sage Weil <sage@redhat.com>

mds: prevent trim count from underflowing

Fixes: http://tracker.ceph.com/issues/21807
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18295 from adamwg/awg/update-orgmap

organizationmap: Add Adam Wolfe Gordon's affiliation

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: change cn mirror to ustc domain

remove cn.ceph.com domain

Signed-off-by: Shengjing Zhu <i@zhsj.me>

Merge PR #18139 into master

* refs/pull/18139/head:
test: make the LibCephFS.ShutdownRacer test even more thrashy
lockdep: free_ids and lock_ref hashes must be truly global
common: add a clear_g_str_vec() function to clear g_str_vec
common: make it safe to call env_to_vec multiple times

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #18183 into master

* refs/pull/18183/head:
osdc/ObjectCacher: limit memory usage of BufferHead

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18215 into master

* refs/pull/18215/head:
mds: update client metadata for already open session

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18261 into master

* refs/pull/18261/head:
mds: keep CInode::STATE_QUEUEDEXPORTPIN state when exporting inode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>