os/bluestore: Do not copy a string just to find its length

I really dislike whoever decided to disallow taking the address of
member functions in the standard library.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

msg: Use with_val, don't copy a string just to call find

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mon: Call get_json_str_map under wth_val

To avoid temporary heap allocations.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mgr: Call _list_modules under with_val

Avoid a temporary heap allocation.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

kv/RocksDB: Do comparisons under with_val

Rather than making a copy on the heap just to compare it.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

kv/RocksDB: Use for_each_substr under with_val

Minimize the number of copies we make.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add with_val to md_config_t

Since passing back references that persist after we drop a lock is
unsafe, allow functions to be called on a reference under the lock.

This way we can compare strings or otherwise examine them without
needing to make a copy.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Disambiguate call to max

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #19244 from ceph/wip-cd-volume

qa/tasks: update ceph-deploy task to use newer ceph-volume syntax

Merge pull request #19276 from tchaikov/wip-volume-key

ceph-volume: do not use --key during mkfs

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19381 from ceph/fix_cephvol_doc

ceph-volume/doc: add missing subcommand in examples

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: add missing subcommand in examples

Add the missing 'lvm' subcommand in examples in the documentation of
ceph-volume.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #18276 from liewegas/wip-removed-snaps

mon,osd,osdc: refactor snap trimming (phase 1)

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #19364 from neha-ojha/wip-preserve-cbt

qa: preserve cbt task results

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: preserve cbt task results

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #19188 from adamwg/awg/rbd-watchers-api

librbd: Add a function to list image watchers

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19337 from trociny/wip-22306

pybind/rbd: fix metadata functions error handling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tests: Add debug info when creating ceph volumes

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #18967 from shun-s/wip-rbd-cp-optimize

librbd: speed up sparse copy when object map is available

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19251 from MahatiC/eventtrace_librbd

librbd: add eventtrace support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19280 from Songweibin/wip-parent-info

rbd: add parent info when moving child into trash bin

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19315 from trociny/wip-fsx-ggate

test/librbd: rbd-ggate mode for fsx

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #18791 into master

* refs/pull/18791/head:
mds: bump protocol for snapshot changes
mds: force client to do snapflush
mds: check existance of snaps when rebuilding need_snapflush
mds: properly setup need_snapflush for flushsnap messages
mds: properly update CInode::first and CDentry first
mds: don't create snap inode during trans-authority rename
mds: trim non-auth snap dentry during cache rejoin
mds: detach unlinked inode's snaprealm from its parent
mds: fix CInode::find_snaprealm()
mds: fix snap dentry discover
mds: introduce MDSMap::get_mds_set_lower_bound()
mds: properly distinguish snaptable TABLE_OP_CREATE from TABLE_OP_UPDATE
mds: properly set reply buffer for table request 'agree' message
mds: make change to mds table after corresponding event get logged

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19351 from ceph/wip-rm22281

ceph-volume rollback on failed OSD prepare/create

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #19344 from tchaikov/wip-we-need-cxx11-and-more

cmake: bail out if GCC version is less than 5.1

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19343 from dillaman/wip-22321

ceph-dencoder: moved RBD types outside of RGW preprocessor guard

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #19333 from amitkumar50/cov-osd-4

osd: Initializing start_offset,last_offset,offset

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #19334 from amitkumar50/cov-misc-13

test: Initializing ChunkReadOp members

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #18849 into master

* refs/pull/18849/head:
mds:improve mds perf counters can statistics filesystem operations number and latency

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18941 into master

* refs/pull/18941/head:
discard the mdsload clear after prep_rebalance in case we want to export it for debugging
make sure that MDBalancer uses heartbeat info from the same epoch

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19089 into master

* refs/pull/19089/head:
client: xlist link dentries instead of set
client: cleanup Dentry definition

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19092 into master

* refs/pull/19092/head:
mds: remove useless check in Migrator::export_dir
mds: ignore export pin for unlinked directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19218 into master

* refs/pull/19218/head:
client: quit on failed remount during dentry invalidate test

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19224 into master

* refs/pull/19224/head:
mds: use find instead of count in CInode::get_dirfrag

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19237 into master

* refs/pull/19237/head:
mds: get rid of the redundant code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19347 from ceph/wip-rm22326

ceph-volume: warn on missing ceph.conf file

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume lvm.create rollback osd creation when prepare or activate fails

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.create fallback to bluestore when objectstore is not specified

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare fallback to bluestore when objectstore is not specified

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare rollback osd when prepare fails and an osd id was generated

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common create a rollback_osd utility to cleanup failed osd prepare/create calls

Signed-off-by: Alfredo Deza <adeza@redhat.com>

pybind/rbd: raise KeyError when metadata does not exist

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

ceph-volume test logging ignored ceph.conf file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume main warn on inability to load ceph.conf, don't raise

Signed-off-by: Alfredo Deza <adeza@redhat.com>

test: Initializing ChunkReadOp members

Fixes the coverity issue:
2. uninit_member: Non-static class member offset is not
initialized in this constructor nor in any functions that it calls.
4. uninit_member: Non-static class member length is not
initialized in this constructor nor in any functions that it calls.

CID 1424433 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
6. uninit_member: Non-static class member tgt_offset is not initialized
in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar amitkuma@redhat.com

osd: Initializing start_offset,last_offset,offset

Fixes the coverity issues:

2. uninit_member: Non-static class member start_offset
is not initialized in this constructor nor in any functions
that it calls.

CID 1424396 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
4. uninit_member: Non-static class member last_offset is not
initialized in this constructor nor in any functions that it calls.

CID 1424658 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member offset is not initialized
in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar <amitkuma@redhat.com>

cmake: bail out if GCC version is less than 5.1

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dencoder: moved RBD types outside of RGW preprocessor guard

Fixes: http://tracker.ceph.com/issues/22321
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/rbd: fix metadata functions error handling

Fixes: http://tracker.ceph.com/issues/22306
Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #19101 from ifed01/wip-ifed-extending-fio

test/fio: extend fio objectstore plugin to better simulate OSD behavior

Merge pull request #18863 from tchaikov/wip-devtoolset-7

build/ops: use devtoolset-7 on centos/rhel-7

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18812 from tchaikov/wip-pybind-rados-service-api

pybind/rados: add rados_service_*()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19307 from tchaikov/wip-drop-ubuntu-trusty

qa,doc: drop support of ubuntu trusty

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rbd: comment/msg cleanup

Signed-off-by: songweibin <song.weibin@zte.com.cn>

rbd: list parent info with "--long" when image child is in trash

We can get child's parent information even it's in
trash bin if "--long" flag is specified.

Signed-off-by: songweibin <song.weibin@zte.com.cn>

mds: bump protocol for snapshot changes

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: force client to do snapflush

Snap inodes that are waiting for flush are auth pinned, they affect
subtree/dirfrag frozen. Force client to do snapflush after waiting
for a period of time.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: check existance of snaps when rebuilding need_snapflush

do nothing if there is no snap

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: properly setup need_snapflush for flushsnap messages

When processing a cap flush message that is re-sent, it's possble
that the sender has already released all WR caps. So can't rely on
CEPH_CAP_ANY_WR check in MDCache:cow_inode().

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: properly update CInode::first and CDentry first

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

install-deps.sh: use gcc-7 on trusty

* always install gcc-7 on trusty
* point g++ to g++-7 if not yet

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec: use devtoolset-7-gcc-c++ on centos/rhel

for better performance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: always use yum-config-manager for consistency

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: install devtoolset-7 for centos/rhel

ready them for gcc7 and libstdc++-7 for better performance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19324 from neha-ojha/wip-fix-collectl

qa: install collectl with cbt task

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19325 from ceph/wip-ceph-disk-man-fix

doc/man remove deprecation of ceph-disk man page title

Reviewed-by: Brad Hubbard

client: quit on failed remount during dentry invalidate test

Fixes: http://tracker.ceph.com/issues/22269
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

client: xlist link dentries instead of set

This saves space and avoids unnecessary set logic. In particular, we no longer
need to do a heap allocation for each Dentry * in the std::set.

Before:

(gdb) print sizeof(Inode)
$1 = 1336
(gdb) print sizeof(Inode::dn_set)
$2 = 48

After:

(gdb) print sizeof(Inode)
$1 = 1360
(gdb) print sizeof(Inode::dentries)
$2 = 24

I'm not sure why the Inode size increased when the member size decreased (weird
padding by g++)? Anyway, we still get the benefit of no heap allocations for
the Dentry *s.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/man remove deprecation of ceph-disk man page title

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa: install collectl with cbt task

Signed-off-by: Neha Ojha <nojha@redhat.com>

test/librbd: rbd-ggate mode for fsx

Right now it works only in lite mode due to ggate kernel driver
does not support resize. Also all operations should be block size
alligned, thus -r and -w options should be properly set.

Example:

  ceph_test_librbd_fsx -G -L -r 512 -w 512 -Z -d -N 1024 rbd test

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

test/librbd: enable fsx lite mode support

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #19241 from ceph/wip-doc-deprecate-cdisk

doc: update references to use ceph-volume

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #19321 from ceph/wip-rm22305

ceph-volume should be able to handle multiple LVM (VG/LV) tags

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #19319 from ceph/wip-rm22297

ceph-volume  handle inline comments in the ceph.conf file

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests.api ensure that we can handle non-ceph tags in lvs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19316 from trociny/wip-blkdev-fixup

common/blkdev: fix build in FreeBSD environment

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

ceph-volume api.lvm only consider 'ceph.' tags in logical volumes

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/install use ceph-volume in manual deployment steps

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados update user-management to use ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados update bluestore-migration to use ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados update add-or-rm-osds to use ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados update bluestore config to use ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados update auth config examples to use ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man add deprecation notices to ceph-disk man page

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: process: disable stdin param of run()

we cannot use process.communicate() to feed the Popen with input,
because, upon return of process.communicate() the stdout,stderr are
closed. see https://docs.python.org/2/library/subprocess.html#subprocess.Popen.communicate .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19275 from qrGitHub/wip-rgw-correct-comment

rgw: correct comment in function parse_credentials

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume configuration allow inlined comments for # and ;

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests verify that INI comments can be inlined

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19314 from ifed01/wip-ifed-affiliation

.githubmap, .mailmap, .organizationmap, .peoplemap: update Igor

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

common/blkdev: fix build in FreeBSD environment

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

.githubmap, .mailmap, .organizationmap, .peoplemap: update Igor
Fedotov's affiliation

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

doc: update docs to remove references of ubuntu 14.04

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/docker-test: drop support for ubuntu 14.04 and centos 6

* we cannot build on centos 6. and we are dropping the support for ubuntu
  14.04 also.
* add ubuntu 17.04: this release is not a LTS release, and added merely for
  demostration purpose.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/hadoop: switch from trusty to ubuntu latest

use the ubuntu_face.yaml facet for the distro used for testing. since we
are dropping the support for trusty.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/fs: use ubuntu_latest for multiclient tasks

instead of using ubuntu 14.04, since we want to drop the support of this
release.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/fs: use ubuntu_latest for libcephfs_java

* instead of using ubuntu 14.04, use ubuntu_latest.since we want
  to drop the support of this release.
* refactor this test to use the facet of ubuntu_latest.yaml.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks: remove test_buildpackages

the buildpackages feature is not part of ceph. and its test was moved to
teuthology in 5a43f8d5. let's remove the leftovers from ceph/ceph. BTW,
we have not run this test suite since then. and its dependency cannot be
fulfilled.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds: don't create snap inode during trans-authority rename

MDCache::cow_inode() may add auth pin to frozen inode

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: trim non-auth snap dentry during cache rejoin

Because corresponding dentry in auth mds may have been pruned (snap deleted)

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>