common: remove already included header files

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

rgw: remove already included header files

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #18752 from liewegas/wip-peering-queue

osd: put peering events in main sharded wq

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #19244 from ceph/wip-cd-volume

qa/tasks: update ceph-deploy task to use newer ceph-volume syntax

Merge pull request #19276 from tchaikov/wip-volume-key

ceph-volume: do not use --key during mkfs

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19381 from ceph/fix_cephvol_doc

ceph-volume/doc: add missing subcommand in examples

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: add missing subcommand in examples

Add the missing 'lvm' subcommand in examples in the documentation of
ceph-volume.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #18276 from liewegas/wip-removed-snaps

mon,osd,osdc: refactor snap trimming (phase 1)

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #19364 from neha-ojha/wip-preserve-cbt

qa: preserve cbt task results

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: preserve cbt task results

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #19188 from adamwg/awg/rbd-watchers-api

librbd: Add a function to list image watchers

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19337 from trociny/wip-22306

pybind/rbd: fix metadata functions error handling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tests: Add debug info when creating ceph volumes

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #18967 from shun-s/wip-rbd-cp-optimize

librbd: speed up sparse copy when object map is available

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19251 from MahatiC/eventtrace_librbd

librbd: add eventtrace support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19280 from Songweibin/wip-parent-info

rbd: add parent info when moving child into trash bin

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19315 from trociny/wip-fsx-ggate

test/librbd: rbd-ggate mode for fsx

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #18791 into master

* refs/pull/18791/head:
mds: bump protocol for snapshot changes
mds: force client to do snapflush
mds: check existance of snaps when rebuilding need_snapflush
mds: properly setup need_snapflush for flushsnap messages
mds: properly update CInode::first and CDentry first
mds: don't create snap inode during trans-authority rename
mds: trim non-auth snap dentry during cache rejoin
mds: detach unlinked inode's snaprealm from its parent
mds: fix CInode::find_snaprealm()
mds: fix snap dentry discover
mds: introduce MDSMap::get_mds_set_lower_bound()
mds: properly distinguish snaptable TABLE_OP_CREATE from TABLE_OP_UPDATE
mds: properly set reply buffer for table request 'agree' message
mds: make change to mds table after corresponding event get logged

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19351 from ceph/wip-rm22281

ceph-volume rollback on failed OSD prepare/create

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #19344 from tchaikov/wip-we-need-cxx11-and-more

cmake: bail out if GCC version is less than 5.1

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19343 from dillaman/wip-22321

ceph-dencoder: moved RBD types outside of RGW preprocessor guard

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #19333 from amitkumar50/cov-osd-4

osd: Initializing start_offset,last_offset,offset

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #19334 from amitkumar50/cov-misc-13

test: Initializing ChunkReadOp members

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #18849 into master

* refs/pull/18849/head:
mds:improve mds perf counters can statistics filesystem operations number and latency

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18941 into master

* refs/pull/18941/head:
discard the mdsload clear after prep_rebalance in case we want to export it for debugging
make sure that MDBalancer uses heartbeat info from the same epoch

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19089 into master

* refs/pull/19089/head:
client: xlist link dentries instead of set
client: cleanup Dentry definition

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19092 into master

* refs/pull/19092/head:
mds: remove useless check in Migrator::export_dir
mds: ignore export pin for unlinked directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19218 into master

* refs/pull/19218/head:
client: quit on failed remount during dentry invalidate test

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19224 into master

* refs/pull/19224/head:
mds: use find instead of count in CInode::get_dirfrag

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19237 into master

* refs/pull/19237/head:
mds: get rid of the redundant code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19347 from ceph/wip-rm22326

ceph-volume: warn on missing ceph.conf file

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume lvm.create rollback osd creation when prepare or activate fails

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.create fallback to bluestore when objectstore is not specified

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare fallback to bluestore when objectstore is not specified

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare rollback osd when prepare fails and an osd id was generated

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common create a rollback_osd utility to cleanup failed osd prepare/create calls

Signed-off-by: Alfredo Deza <adeza@redhat.com>

pybind/rbd: raise KeyError when metadata does not exist

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

ceph-volume test logging ignored ceph.conf file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume main warn on inability to load ceph.conf, don't raise

Signed-off-by: Alfredo Deza <adeza@redhat.com>

test: Initializing ChunkReadOp members

Fixes the coverity issue:
2. uninit_member: Non-static class member offset is not
initialized in this constructor nor in any functions that it calls.
4. uninit_member: Non-static class member length is not
initialized in this constructor nor in any functions that it calls.

CID 1424433 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
6. uninit_member: Non-static class member tgt_offset is not initialized
in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar amitkuma@redhat.com

osd: Initializing start_offset,last_offset,offset

Fixes the coverity issues:

2. uninit_member: Non-static class member start_offset
is not initialized in this constructor nor in any functions
that it calls.

CID 1424396 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
4. uninit_member: Non-static class member last_offset is not
initialized in this constructor nor in any functions that it calls.

CID 1424658 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member offset is not initialized
in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar <amitkuma@redhat.com>

cmake: bail out if GCC version is less than 5.1

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dencoder: moved RBD types outside of RGW preprocessor guard

Fixes: http://tracker.ceph.com/issues/22321
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/rbd: fix metadata functions error handling

Fixes: http://tracker.ceph.com/issues/22306
Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #19101 from ifed01/wip-ifed-extending-fio

test/fio: extend fio objectstore plugin to better simulate OSD behavior

Merge pull request #18863 from tchaikov/wip-devtoolset-7

build/ops: use devtoolset-7 on centos/rhel-7

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18812 from tchaikov/wip-pybind-rados-service-api

pybind/rados: add rados_service_*()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19307 from tchaikov/wip-drop-ubuntu-trusty

qa,doc: drop support of ubuntu trusty

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rbd: comment/msg cleanup

Signed-off-by: songweibin <song.weibin@zte.com.cn>

rbd: list parent info with "--long" when image child is in trash

We can get child's parent information even it's in
trash bin if "--long" flag is specified.

Signed-off-by: songweibin <song.weibin@zte.com.cn>

mds: bump protocol for snapshot changes

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: force client to do snapflush

Snap inodes that are waiting for flush are auth pinned, they affect
subtree/dirfrag frozen. Force client to do snapflush after waiting
for a period of time.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: check existance of snaps when rebuilding need_snapflush

do nothing if there is no snap

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: properly setup need_snapflush for flushsnap messages

When processing a cap flush message that is re-sent, it's possble
that the sender has already released all WR caps. So can't rely on
CEPH_CAP_ANY_WR check in MDCache:cow_inode().

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: properly update CInode::first and CDentry first

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

install-deps.sh: use gcc-7 on trusty

* always install gcc-7 on trusty
* point g++ to g++-7 if not yet

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec: use devtoolset-7-gcc-c++ on centos/rhel

for better performance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: always use yum-config-manager for consistency

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: install devtoolset-7 for centos/rhel

ready them for gcc7 and libstdc++-7 for better performance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19324 from neha-ojha/wip-fix-collectl

qa: install collectl with cbt task

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19325 from ceph/wip-ceph-disk-man-fix

doc/man remove deprecation of ceph-disk man page title

Reviewed-by: Brad Hubbard

client: quit on failed remount during dentry invalidate test

Fixes: http://tracker.ceph.com/issues/22269
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

client: xlist link dentries instead of set

This saves space and avoids unnecessary set logic. In particular, we no longer
need to do a heap allocation for each Dentry * in the std::set.

Before:

(gdb) print sizeof(Inode)
$1 = 1336
(gdb) print sizeof(Inode::dn_set)
$2 = 48

After:

(gdb) print sizeof(Inode)
$1 = 1360
(gdb) print sizeof(Inode::dentries)
$2 = 24

I'm not sure why the Inode size increased when the member size decreased (weird
padding by g++)? Anyway, we still get the benefit of no heap allocations for
the Dentry *s.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/man remove deprecation of ceph-disk man page title

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa: install collectl with cbt task

Signed-off-by: Neha Ojha <nojha@redhat.com>

osd/PGBackend: remove unused on_flushed()

Signed-off-by: Sage Weil <sage@redhat.com>

osd: teach mclock about peering_event

Signed-off-by: Sage Weil <sage@redhat.com>

osd: wait for pgs to catch up in handle_osd_map

Make sure we don't inject maps too far out in front of the PGs ability
to consume them.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: keep maps pinned in cache that pgs haven't consumed yet

This ensures that we do not drop maps from the map cache that PGs
are still trying to consume.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove obsolete osd_map_max_advance option

We now always advance to the latest map epoch at once.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: be a bit more pedantic about pg epoch

Make sure PG has activated it before we mark it fully consumed.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop max behavior, return value in advance_pg

This is no longer needed/used.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove old peering work queue

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_workqueue: use different option

Signed-off-by: Sage Weil <sage@redhat.com>

osd: change peering wq "drain" while not active

Use the min epoch helpers instead of explicitly draining the work
queue to make sure that app PGs have processed the new maps.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move peering events into primary work queue

For the moment we leave the old infratructure in place too; that will be
cleaned up next.

Two key differences:

- each pg event is processed in isolation, which means MOSDPGNotify etc
messages won't be batched.  This will mean more messages for small clusters
and no change for large clusters.
- the map processing for PGs will be done in completion by a single event.
There is now a wait in handle_osd_map to ensure that all PGs are processing
maps that overlap with the map cache.

Signed-off-by: Sage Weil <sage@redhat.com>

test/librbd: rbd-ggate mode for fsx

Right now it works only in lite mode due to ggate kernel driver
does not support resize. Also all operations should be block size
alligned, thus -r and -w options should be properly set.

Example:

  ceph_test_librbd_fsx -G -L -r 512 -w 512 -Z -d -N 1024 rbd test

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

test/librbd: enable fsx lite mode support

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #19241 from ceph/wip-doc-deprecate-cdisk

doc: update references to use ceph-volume

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #19321 from ceph/wip-rm22305

ceph-volume should be able to handle multiple LVM (VG/LV) tags

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #19319 from ceph/wip-rm22297

ceph-volume  handle inline comments in the ceph.conf file

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests.api ensure that we can handle non-ceph tags in lvs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19316 from trociny/wip-blkdev-fixup

common/blkdev: fix build in FreeBSD environment

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

ceph-volume api.lvm only consider 'ceph.' tags in logical volumes

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/install use ceph-volume in manual deployment steps

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados update user-management to use ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados update bluestore-migration to use ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados update add-or-rm-osds to use ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados update bluestore config to use ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados update auth config examples to use ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man add deprecation notices to ceph-disk man page

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: process: disable stdin param of run()

we cannot use process.communicate() to feed the Popen with input,
because, upon return of process.communicate() the stdout,stderr are
closed. see https://docs.python.org/2/library/subprocess.html#subprocess.Popen.communicate .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19275 from qrGitHub/wip-rgw-correct-comment

rgw: correct comment in function parse_credentials

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume configuration allow inlined comments for # and ;

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests verify that INI comments can be inlined

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19314 from ifed01/wip-ifed-affiliation

.githubmap, .mailmap, .organizationmap, .peoplemap: update Igor

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

common/blkdev: fix build in FreeBSD environment

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

.githubmap, .mailmap, .organizationmap, .peoplemap: update Igor
Fedotov's affiliation

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

doc: update docs to remove references of ubuntu 14.04

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/docker-test: drop support for ubuntu 14.04 and centos 6

* we cannot build on centos 6. and we are dropping the support for ubuntu
  14.04 also.
* add ubuntu 17.04: this release is not a LTS release, and added merely for
  demostration purpose.

Signed-off-by: Kefu Chai <kchai@redhat.com>