ceph-volume: exceptions: create a module for all exceptions

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: systemd: move the exceptions to the new exception module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: decorators: add a check for super user privileges

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: main: use subhelp to parse lvm's help

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: terminal: use raw instead of write

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: main: no need to pass the config to log setup

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: log: notify to stdout if it is not possible to write to the log location

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: systemd: create systemd entry point script

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: process: module for running system commands

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: main: initial take on main

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: terminal: easier terminal reporting utils

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: decorators: catch exceptions, disable on debug

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: log: be more robust, report back to config

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: log: create a utility for setting up logging

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: config: default to 'info' verbosity, set the config dict

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: rename: initial take on renaming to ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

cmake/Distutils.cmake: make pkg files optional, add bin

Skip nonexistent 'expected' Python packaging files, and add bin/ as
a possible file for symlinking during build (for Python projects with
slightly different forms, currently ceph-volume)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

build: include the ceph-volume-systemd command in the spec file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

build: add ceph-volume to CMakeLists in src

Signed-off-by: Alfredo Deza <adeza@redhat.com>

systemd: include the ceph-volume service

Signed-off-by: Alfredo Deza <adeza@redhat.com>

build: create a CMakeLists file for ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

systemd: create a service file for ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

build: add deb install rule for ceph-volume executable

Signed-off-by: Alfredo Deza <adeza@redhat.com>

build: add deb rules for ceph-volume systemd service file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

build/ops: add ceph-volume scripts/files to spec file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: create the executables

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: packaging: create a setup.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: tests: create a tox.ini file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: add a manifest.in file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: create a .gitignore specific to ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #16805 from liewegas/wip-crush-classes

crush: misc changes/fixes for device classes

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16739 from liewegas/wip-multi-backfill-reject

qa/suites/rados/singleton-nomsgr/all/multi-backfill-reject: sleep longer

Merge pull request #16794 from liewegas/wip-rbd-app

qa/tasks/ceph: enable rbd on rbd pool

Merge pull request #16776 from dachary/wip-20893-ceph-fsid

ceph-disk: dmcrypt cluster must default to ceph

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: enable rbd on rbd pool

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16798 from jdurgin/wip-complete-to-loop

osd/PGLog: avoid infinite loop if missing version is corrupted

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16804 from xiexingguo/wip-some-cleanup

os/bluestore: space between func and contents

qa/suites/rados/thrash-erasure-code-big: add k=4 m=2

Get better coverage for larger codes.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: space between func and contents

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

crush: remove cleanup_classes()

I can't for the life of me figure out what this is supposed to do.

- why remove (some) classes right after we populated them?
- why remove them after we decode the crush map?

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: rebuild_roots_with_classes on bucket removal

Signed-off-by: Sage Weil <sage@redhat.com>

crush: fix preservation of shadow bucket ids

1- a decompiled and recompiled was parsing the class bucket ids but it
wasn't actually using them.
2- rebuild_roots_with_classes() was throwing out the old ids and assigning
new ids when the tree was rebuilt.

Fix by passing in a (potentially partial) class_bucket map into
populate_classes().  Take care to allocate new bucket ids that don't
collide with previously used ids.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #16802 into master

* refs/remotes/upstream/pull/16802/head:
qa: update wait_for_health for new health json syntax

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: update wait_for_health for new health json syntax

Fixes: http://tracker.ceph.com/issues/20890
Signed-off-by: John Spray <john.spray@redhat.com>

Merge PR #16766 into master

* refs/remotes/upstream/pull/16766/head:
cephfs: Clarify health warning
cephfs: Always print filesystem name

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs: Clarify health warning

Add text to MDS_INSUFFICIENT STANDBY to clarify it refers
to MDS daemons.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

cephfs: Always print filesystem name

Previously, the FSMap summary omitted the filesystem
name in clusters with only one filesystem. Print the
file system name no matter what.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

qa/workunits/mon/crush_ops.sh: remove existing dev class before setting it

we cannot overwrite existing dev class, and "osd_class_update_on_start"
is true by default (see 0c885d6). so we should remove all device classes before
setting them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: kill unused variable 'ts'

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/osd-fast-mark-down: enable 'osd-class-update-on-start' by default

https://github.com/ceph/ceph/pull/16016/commits/116cf759c86ca9ca7d80769b2096953516f357cb
will now hide all shadow trees(roots), so this is not applicable anymore
(actually it is misleading).

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

crush: make 'remove_root' idempotent

We might use 'crush link' to link same host into
different roots, which as a result can cause different
shadow trees reference same hosts too.

This means we may need to destory the same buckets(hosts, racks, etc.)
multiple times during rebuilding all shadow trees and hence 'remove_root'
shall be idempotent.

Fixes: http://tracker.ceph.com/issues/20845
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16740 from liewegas/wip-crush-tree

mon: show class in 'osd crush tree' output; sort output

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #16772 into master

* refs/remotes/upstream/pull/16772/head:
qa: add health whitelist for all fs sub-suites

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add health whitelist for all fs sub-suites

Fixes: http://tracker.ceph.com/issues/20892
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #16768 into master

* refs/remotes/upstream/pull/16768/head:
qa: fix log whitelist string

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #16743 into master

* refs/remotes/upstream/pull/16743/head:
mds: add inter-mds operations to PerfCounter

Merge PR #16654 into master

* refs/remotes/upstream/pull/16654/head:
mds: fix race in "open inode by inode number"
mds: cleanup {do_open,_do_find}_ino_peer()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #16631 into master

* refs/remotes/upstream/pull/16631/head:
cephfs: Resend discover when joining from starting state
cephfs: Wait for MDCache to be ready before starting log segment

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #16363 into master

* refs/remotes/upstream/pull/16363/head:
mds: fix health checks for multiple filesystems
mon: fix log message on mds rank stop
mon: uniform way of referring to MDS daemons
mds: revise health checks for _FAILED/_DEGRADED
mon/MDSMonitor: log messages on MDS rank changes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #16292 into master

* refs/remotes/upstream/pull/16292/head:
qa: use new hex rep of inode
qa: fix whitelist error message
mds: refine "Scrub error" cluster log message
mds: polish clog messages
doc: developer logging guidance

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #16749 from tchaikov/wip-restful-delete-key

mgr: handle "module.set_config(.., None)" correctly

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #16756 from liewegas/wip-20113

ceph: wait for maps before doing 'ceph tell ... help'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16760 from cbodley/wip-rgw-disable-lifecycle-s3tests

qa/rgw: disable lifecycle tests because of expiration failures

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #16491 from cbodley/wip-20308

rgw: match wildcards in StringLike policy conditions

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

ceph-disk: dmcrypt cluster must default to ceph

If ceph_fsid is not found, which is the case for legacy dmcrypted OSD,
the cluster must default to ceph, as it was before support for non
standard cluster names was introduced.

Fixes: http://tracker.ceph.com/issues/20893
Signed-off-by: Loic Dachary <loic@dachary.org>

osd/PGLog: avoid infinite loop if missing version is corrupted

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16738 from liewegas/wip-20810

os/bluestore: fix dirty_range on _do_clone_range

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16789 from liewegas/wip-ec-m-2

qa: avoid map-gap tests for k=2 m=1

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16795 from liewegas/wip-mgr-whitelist

qa/suites/rados/mgr/tasks/failover: whitelist

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/thrash-erasure-code: do not test map gap with m=1

We test EC profiles with m=1 here, and mapgap can lead to incomplete pgs
because it takes an osd down and waits for healthy.

Fixes: http://tracker.ceph.com/issues/20844
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-erasure-coe-big/clsuter: 12 osds on 3 nodes not 4

smithi have 4 nvme partitions available, not 3.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16797 from jdurgin/wip-upgrade-jewel-x

qa: timeout when waiting for mgr to be available in healthy()

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-erasure-code-big: add k=4 m=2

Get better coverage for larger codes.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/monthrash/worklaods/rados_api_tests: whitelist SMALLER_PGP_NUM

The rados/test.sh fiddles with pg_num.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/mgr/tasks/failover: whitelist

remote/smithi025/log/ceph.log.gz:2017-08-03 07:02:15.049074 mon.b mon.0 172.21.15.25:6789/0 197 : cluster [INF] Manager daemon x is unresponsive, replacing it with standby daemon y
remote/smithi025/log/ceph.log.gz:2017-08-03 07:03:10.078032 mon.b mon.0 172.21.15.25:6789/0 226 : cluster [WRN] Manager daemon x is unresponsive.  No standby daemons available.

x and y may be swapped, so whitelist the rest of the string.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16764 from liewegas/wip-release-notes

doc/release-notes: sort release note changes into the right section

Merge pull request #16765 from liewegas/wip-bluestore-docs

doc/rados/configuration: document bluestore

doc/rados/configuration/bluestore-config-ref: devices, checksumming, cache

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16773 from dmick/wip-make-check-cleanup

Clean up make check for persistent test nodes (like arm64)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16683 from Liuchang0812/clear-result-in-getchildren

os/bluestore: clear result in BlueRocksEnv::getChildren

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16717 from gmayyyha/osd-objectstore-20865

ceph-osd: fix auto detect which objectstore is currently running

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #16777 from SUSE/wip-20894

doc: document "rados cleanup" in rados manpage

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16775 from ovh/bp-fr-pendingnotes-update

PendingReleaseNotes: mention forced recovery

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: document "rados cleanup" in rados manpage

Fixes: http://tracker.ceph.com/issues/20894
Signed-off-by: Nathan Cutler <ncutler@suse.com>

PendingReleaseNotes: mention forced recovery

Mention forced recovery in release notes.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

qa: use new hex rep of inode

Resolves a failure from QA:

    2017-08-02T19:23:27.489 INFO:tasks.cephfs_test_runner:======================================================================
    2017-08-02T19:23:27.489 INFO:tasks.cephfs_test_runner:FAIL: test_oversize (tasks.cephfs.test_fragment.TestFragmentation)
    2017-08-02T19:23:27.489 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
    2017-08-02T19:23:27.490 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
    2017-08-02T19:23:27.490 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20170802/qa/tasks/cephfs/test_fragment.py", line 71, in test_oversize
    2017-08-02T19:23:27.490 INFO:tasks.cephfs_test_runner:    self.assertEqual(frags[0]['dirfrag'], "10000000000.0*")
    2017-08-02T19:23:27.490 INFO:tasks.cephfs_test_runner:AssertionError: u'0x10000000000.0*' != '10000000000.0*'
    2017-08-02T19:23:27.490 INFO:tasks.cephfs_test_runner:

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

test/encoding/readable.sh: clean up all tmpfiles/dirs

Signed-off-by: Dan Mick <dan.mick@redhat.com>

run-make-check.sh: set ulimit -n (because no one else does)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #16678 from mychoxin/yuanxin_make_print_informative

print more information when run ceph-osd cmd with 'check options'

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #16713 into master

* refs/remotes/upstream/pull/16713/head:
qa: ignore failed MDS message during upgrade

Merge PR #16691 into master

* refs/remotes/upstream/pull/16691/head:
doc: update client.admin key refs to "mds 'allow'"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #16714 into master

* refs/remotes/upstream/pull/16714/head:
qa: test export_pin is correct in dumped subtree
mds: print export_pin for dumped subtree

Reviewed-by: Douglas Fuller <dfuller@redhat.com>
Reviewed-by: huanwen ren <ren.huanwen@zte.com.cn>

qa: fix whitelist error message

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: fix log whitelist string

Fixes: http://tracker.ceph.com/issues/20889
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #16732 from Abhishekvrshny/fix-rgw-mp-races

rgw : fix race in RGWCompleteMultipart

doc/rados/configuration: document bluestore

Initial pass here.  Not yet complete.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

rgw : fix race in RGWCompleteMultipart

Fixes : http://tracker.ceph.com/issues/20861

Signed-off-by: Abhishek Varshney <abhishek.varshney@flipkart.com>

Merge pull request #16377 from theanalyst/doc/release-notes/12.1.1

doc: 12.1.1 & 12.1.2 release notes

Merge pull request #16308 from joscollin/wip-rbd-nbd-return-values

tools/rbd, rgw: Removed unreachable returns

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>