ceph-volume: init: define a few conf attributes

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: process: do not block logging on subprocess prompts

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: main: store ceph conf and load it in the conf module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: configuration: create a module to load/parse ceph configs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: log: use new config module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: init: define the configuration modules

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: exceptions: add configuration errors

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: lvm.activate: udpdate the help to normalize representations

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: exceptions: add error for multiple volume groups

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: lvm.activate: add object store type support for the flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: lvm.api: support volume groups with same API as lvs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: lvm.prepare: initial take on the module with flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: lvm.api return None when .get() doesn't match anything

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: lvm activate checks other non-lv journals and symlinks accordingly

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: lvm create the functional activate module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: lvm create an api module with better interfaces/helpers

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: systemd: create a systemctl module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: systemd: move to directory

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: systemd: create an actual directory module to add more related modules in

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: process: add a 'call' function helper, similar to subprocess.call

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: main: remove extra new lines when generating help

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: exceptions: create a MultipleLV error

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: lvm: initial take on tag api

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: exceptions: create a module for all exceptions

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: systemd: move the exceptions to the new exception module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: decorators: add a check for super user privileges

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: main: use subhelp to parse lvm's help

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: terminal: use raw instead of write

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: main: no need to pass the config to log setup

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: log: notify to stdout if it is not possible to write to the log location

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: systemd: create systemd entry point script

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: process: module for running system commands

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: main: initial take on main

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: terminal: easier terminal reporting utils

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: decorators: catch exceptions, disable on debug

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: log: be more robust, report back to config

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: log: create a utility for setting up logging

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: config: default to 'info' verbosity, set the config dict

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: rename: initial take on renaming to ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

cmake/Distutils.cmake: make pkg files optional, add bin

Skip nonexistent 'expected' Python packaging files, and add bin/ as
a possible file for symlinking during build (for Python projects with
slightly different forms, currently ceph-volume)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

build: include the ceph-volume-systemd command in the spec file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

build: add ceph-volume to CMakeLists in src

Signed-off-by: Alfredo Deza <adeza@redhat.com>

systemd: include the ceph-volume service

Signed-off-by: Alfredo Deza <adeza@redhat.com>

build: create a CMakeLists file for ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

systemd: create a service file for ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

build: add deb install rule for ceph-volume executable

Signed-off-by: Alfredo Deza <adeza@redhat.com>

build: add deb rules for ceph-volume systemd service file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

build/ops: add ceph-volume scripts/files to spec file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: create the executables

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: packaging: create a setup.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: tests: create a tox.ini file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: add a manifest.in file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: create a .gitignore specific to ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #16805 from liewegas/wip-crush-classes

crush: misc changes/fixes for device classes

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16739 from liewegas/wip-multi-backfill-reject

qa/suites/rados/singleton-nomsgr/all/multi-backfill-reject: sleep longer

Merge pull request #16794 from liewegas/wip-rbd-app

qa/tasks/ceph: enable rbd on rbd pool

Merge pull request #16776 from dachary/wip-20893-ceph-fsid

ceph-disk: dmcrypt cluster must default to ceph

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: enable rbd on rbd pool

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16798 from jdurgin/wip-complete-to-loop

osd/PGLog: avoid infinite loop if missing version is corrupted

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16804 from xiexingguo/wip-some-cleanup

os/bluestore: space between func and contents

qa/suites/rados/thrash-erasure-code-big: add k=4 m=2

Get better coverage for larger codes.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: space between func and contents

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

crush: remove cleanup_classes()

I can't for the life of me figure out what this is supposed to do.

- why remove (some) classes right after we populated them?
- why remove them after we decode the crush map?

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: rebuild_roots_with_classes on bucket removal

Signed-off-by: Sage Weil <sage@redhat.com>

crush: fix preservation of shadow bucket ids

1- a decompiled and recompiled was parsing the class bucket ids but it
wasn't actually using them.
2- rebuild_roots_with_classes() was throwing out the old ids and assigning
new ids when the tree was rebuilt.

Fix by passing in a (potentially partial) class_bucket map into
populate_classes().  Take care to allocate new bucket ids that don't
collide with previously used ids.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #16802 into master

* refs/remotes/upstream/pull/16802/head:
qa: update wait_for_health for new health json syntax

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: update wait_for_health for new health json syntax

Fixes: http://tracker.ceph.com/issues/20890
Signed-off-by: John Spray <john.spray@redhat.com>

Merge PR #16766 into master

* refs/remotes/upstream/pull/16766/head:
cephfs: Clarify health warning
cephfs: Always print filesystem name

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs: Clarify health warning

Add text to MDS_INSUFFICIENT STANDBY to clarify it refers
to MDS daemons.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

cephfs: Always print filesystem name

Previously, the FSMap summary omitted the filesystem
name in clusters with only one filesystem. Print the
file system name no matter what.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

qa/workunits/mon/crush_ops.sh: remove existing dev class before setting it

we cannot overwrite existing dev class, and "osd_class_update_on_start"
is true by default (see 0c885d6). so we should remove all device classes before
setting them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: kill unused variable 'ts'

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/osd-fast-mark-down: enable 'osd-class-update-on-start' by default

https://github.com/ceph/ceph/pull/16016/commits/116cf759c86ca9ca7d80769b2096953516f357cb
will now hide all shadow trees(roots), so this is not applicable anymore
(actually it is misleading).

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

crush: make 'remove_root' idempotent

We might use 'crush link' to link same host into
different roots, which as a result can cause different
shadow trees reference same hosts too.

This means we may need to destory the same buckets(hosts, racks, etc.)
multiple times during rebuilding all shadow trees and hence 'remove_root'
shall be idempotent.

Fixes: http://tracker.ceph.com/issues/20845
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16740 from liewegas/wip-crush-tree

mon: show class in 'osd crush tree' output; sort output

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #16772 into master

* refs/remotes/upstream/pull/16772/head:
qa: add health whitelist for all fs sub-suites

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add health whitelist for all fs sub-suites

Fixes: http://tracker.ceph.com/issues/20892
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #16768 into master

* refs/remotes/upstream/pull/16768/head:
qa: fix log whitelist string

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #16743 into master

* refs/remotes/upstream/pull/16743/head:
mds: add inter-mds operations to PerfCounter

Merge PR #16654 into master

* refs/remotes/upstream/pull/16654/head:
mds: fix race in "open inode by inode number"
mds: cleanup {do_open,_do_find}_ino_peer()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #16631 into master

* refs/remotes/upstream/pull/16631/head:
cephfs: Resend discover when joining from starting state
cephfs: Wait for MDCache to be ready before starting log segment

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #16363 into master

* refs/remotes/upstream/pull/16363/head:
mds: fix health checks for multiple filesystems
mon: fix log message on mds rank stop
mon: uniform way of referring to MDS daemons
mds: revise health checks for _FAILED/_DEGRADED
mon/MDSMonitor: log messages on MDS rank changes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #16292 into master

* refs/remotes/upstream/pull/16292/head:
qa: use new hex rep of inode
qa: fix whitelist error message
mds: refine "Scrub error" cluster log message
mds: polish clog messages
doc: developer logging guidance

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #16749 from tchaikov/wip-restful-delete-key

mgr: handle "module.set_config(.., None)" correctly

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #16756 from liewegas/wip-20113

ceph: wait for maps before doing 'ceph tell ... help'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16760 from cbodley/wip-rgw-disable-lifecycle-s3tests

qa/rgw: disable lifecycle tests because of expiration failures

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #16491 from cbodley/wip-20308

rgw: match wildcards in StringLike policy conditions

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

ceph-disk: dmcrypt cluster must default to ceph

If ceph_fsid is not found, which is the case for legacy dmcrypted OSD,
the cluster must default to ceph, as it was before support for non
standard cluster names was introduced.

Fixes: http://tracker.ceph.com/issues/20893
Signed-off-by: Loic Dachary <loic@dachary.org>

osd/PGLog: avoid infinite loop if missing version is corrupted

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16738 from liewegas/wip-20810

os/bluestore: fix dirty_range on _do_clone_range

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16789 from liewegas/wip-ec-m-2

qa: avoid map-gap tests for k=2 m=1

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16795 from liewegas/wip-mgr-whitelist

qa/suites/rados/mgr/tasks/failover: whitelist

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/thrash-erasure-code: do not test map gap with m=1

We test EC profiles with m=1 here, and mapgap can lead to incomplete pgs
because it takes an osd down and waits for healthy.

Fixes: http://tracker.ceph.com/issues/20844
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-erasure-coe-big/clsuter: 12 osds on 3 nodes not 4

smithi have 4 nvme partitions available, not 3.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16797 from jdurgin/wip-upgrade-jewel-x

qa: timeout when waiting for mgr to be available in healthy()

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-erasure-code-big: add k=4 m=2

Get better coverage for larger codes.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/monthrash/worklaods/rados_api_tests: whitelist SMALLER_PGP_NUM

The rados/test.sh fiddles with pg_num.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/mgr/tasks/failover: whitelist

remote/smithi025/log/ceph.log.gz:2017-08-03 07:02:15.049074 mon.b mon.0 172.21.15.25:6789/0 197 : cluster [INF] Manager daemon x is unresponsive, replacing it with standby daemon y
remote/smithi025/log/ceph.log.gz:2017-08-03 07:03:10.078032 mon.b mon.0 172.21.15.25:6789/0 226 : cluster [WRN] Manager daemon x is unresponsive.  No standby daemons available.

x and y may be swapped, so whitelist the rest of the string.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16764 from liewegas/wip-release-notes

doc/release-notes: sort release note changes into the right section

Merge pull request #16765 from liewegas/wip-bluestore-docs

doc/rados/configuration: document bluestore