ceph-disk: cast output of subprocess.Popen() to str()

Cast output of subprocess.Popen() to str() to be able to use
str.split() and str.splitlines() without warnings from pylint.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: fix adjust_symlink() replace 'journal' with 'target'

Replace 'journal' variable with 'target' since journal doesn't
exist in this function.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: fix adjust_symlink() replace 'canonical' with 'path'

Replace 'canonical' variable with 'path' since canonical doesn't
exist in this function.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: there is no os.path.lstat use os.lstat

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: remove unused variable key from prepare_journal_dev()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: fix except to catch OSError

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: add missing space after >> operator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

fix: Redefining name 'uuid' from outer scope (line 14)

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: remove unused variables from list_partitions()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: fix /dev/dm-[0-9] handling list_all_partitions()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #119 from dalgaaf/wip-da-ceph-disk

Cleanup and fixes for wip-ceph-disk

ceph-disk: rename local variable shadowing builtin

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: remove twice defined identical function unmount

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: remove twice defined function mount

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: remove double defined function get_conf

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: rename local variable shadowing builtin

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: install and package

Signed-off-by: Sage Weil <sage@inktank.com>

ceph-disk: simplify command dispatch

Signed-off-by: Sage Weil <sage@inktank.com>

ceph-disk: consolidate exceptions

Use a single exception type, and catch it at the top level.

Signed-off-by: Sage Weil <sage@inktank.com>

ceph-disk: consolidate ceph-disk-* into a single binary

 ceph-disk prepare ...
 ceph-disk activate ...
 ceph-disk ...

This let's us share code (we were already duplicating a bunch of stuff!)
and to add new commands easily.

Signed-off-by: Sage Weil <sage@inktank.com>

PendingReleaseNotes: fix typo

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

librbd: optionally wait for a flush before enabling writeback

Older guests may not send flushes properly (i.e. never), so if this is
enabled, rbd_cache=true is safe for them transparently.

Disable by default, since it will unnecessarily slow down newer guest
boot, and prevent writeback caching for things that don't need to send
flushes, like the command line tool.

Refs: #3817
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

Makefile: missing header

Signed-off-by: Sage Weil <sage@inktank.com>

mon: use enum instead of static const int

This way it compiles.

Signed-off-by: Sage Weil <sage@inktank.com>

Makefile.am: fix misspelt header name

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge branch 'next'

ceph-disk-prepare: 'mkfs -t' instead of 'mkfs --type='

Older mkfs (el6) doesn't like --type=.

Fixes: #4495
Reported-by: Alexandre Maragone <alexandre.maragone@inktank.com>
Signed-off-by: Sage Weil <sage@inktank.com>

mds: Handle ENODATA returned from getxattr

The osds might return ENODATA if we request an
xattr that doesn't exist.  In this case, we're
requesting the 'parent' xattr so that we can
remove all the forwarding pointers, but the xattr
may not have been written (which only happens on
log segment trim), so we don't assert here.

Signed-off-by: Sam Lang <sam.lang@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

mon/PaxosService: fix proposal waiter handling

- Cancel the propsal waiters with EAGAIN on election, etc.
- Drop the wakeup helper and open-code the one caller.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

Makefile: fix header name

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #115 from ceph/wip-4199

Resolves #4199

Reviewed-by: Sage Weil <sage@inktank.com>

mon: Monitor: take advantage of the new HealthMonitor class.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: HealthMonitor: Keep track of monitor cluster's health

The HealthMonitor builds upon the QuorumService interface, and should be
used to keep track of all and any relevant information about the monitor
cluster (maybe even about all the cluster if need be).

This patch also introduces the HealthService interface, used to define
a HealthMonitor service, responsible for dispatching 'MMonHealth' messages
(the QuorumService interface dispatches generic 'Message').

Based on the HealthService interface, we introduce the DataHealthService
class, a service that will track disk space consumption by the monitors,
warn when a given threshold is crossed, and gracefully shutdown the monitor
if disk space usage hits critical levels that might affect the correct
monitor behavior.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: QuorumService: Allow for services quorum-bound to be easily created

As the monitor grows in features, we have been dumping them in the Monitor
class as they don't really fit anywhere else.

Most of those latest features have been, and some of the future changes
will also be, quorum-bounded.  By that we mean that these features tend
to require a quorum to be present in order to work.

Although we already have the PaxosService interface, it really isn't
adequate for this kind of features, as they don't really require Paxos,
nor do they access the store.  Furthermore, they don't really need to
tick at the same rate as the monitor, and can be fairly independent.

Therefore we now introduce the concept of a QuorumService, a class to be
built upon, managing the tick and dispatch for any kind of service
basically requiring a quorum to function.

Among the already existing monitor features that could take advantage of
this new class we can find the Timecheck infrastructure, as it is by
nature quorum bounded.  The monitor store sync could also take advantage
of this service, although it doesn't really require a quorum to work,
and even the PaxosService-related classes could use this.

This patch also introduces the MMonQuorumService base class, to be used
by any message that should want to.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa: add 16MB direct-io write test

See bug #2657.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #113 from dalgaaf/wip-da-performance-3

Iterator related fixes

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #112 from dalgaaf/wip-da-performance-2-v2

prefer prefix ++/--operator for e.g. iterators for performance reasons

Reviewed-by: Sage Weil <sage@inktank.com>

signal_handler: add func to queue an async signal

For example, queue a SIGINT without actually sending the signal via the
OS.

Signed-off-by: Sage Weil <sage@inktank.com>

client:  Remove unecessary set_inode() in _rmdir()

With the recent changes in fc80c1dc6ee315ae5e039986602ffadba46cb43b,
we only allow setting the inode once on a MetaRequest.  This triggered
a bug in _rmdir(), where the parent dir inode passed in and being set
on the MetaRequest, and then also setting the dir inode on the MetaRequest.
Removing the set_inode() using the parent dir inode resolves this issue.

Signed-off-by: Sam Lang <sam.lang@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

mon/Session.h: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/filestore/workload_generator.cc: prefer prefix ++operator for iterators

Prefer prefix ++operator for iterator. Move the iterator declaration into
the for-loop header since the iterator isn't used outside the loop.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/filestore/test_idempotent.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/filestore/test_idempotent.cc: fix ~TestFileStoreState()

Fix interator handling in ~TestFileStoreState(). After std::map::erase()
the used iterator is invalid. Use a while-loop and erase the object with
post-incremented iterator instead.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/test_snap_mapper.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/osd/RadosModel.h: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/filestore/TestFileStoreState.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/xattr_bench.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/osd/Object.cc: prefer prefix --/++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/encoding/ceph_dencoder.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/bench/bencher.h: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ObjectMap/test_object_map.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

tools/rest_bench.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

tools/common.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-filestore-dump.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_user.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_rest_swift.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/OSDMap.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/MDCache.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/CInode.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

client/Client.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

auth/AuthMethodList.cc: fix remove_supported_auth()

Make sure the iterator isn't incremented twice if a element gets erased.
Move increment out of the for-loop header and add it to a new else block.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/mdstypes.h: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_rados.cc: make sure range_iter != ranges.end()

Make sure range_iter is valid, set range_iter = next_iter instead of
++range_iter, since next_iter is already checked against ranges.end() and
is the same as ++range_iter.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados.cc: prefer prefix ++operator for iterators

Prefer prefix ++operator for iterators, move some iterator declaration into
the for-loop header, since the iterator is only used within the loop.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_rados.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_admin.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_acl_s3.cc: prefer prefix ++operator for iterators

Prefer prefix ++operator for iterators, move some iterator declaration into
the for-loop header, since the iterator is only used within the loops.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osdmaptool.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osdc/Striper.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osdc/Objecter.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osdc/ObjectCacher.cc: prefer prefix ++/--operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osdc/ObjectCacher.h: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osdc/Objecter.h: prefer prefix ++operator for iterators

Prefer prefix ++operator for iterators, move one iterator declaration into
the for-loop header, since the iterator is only used within the loop.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osdc/Filer.h: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osdc/Filer.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/osd_types.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/SnapMapper.cc: use empty() instead of size() == 0

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/ReplicatedPG.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/PG.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/OSDMap.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/OSD.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

os/LFNIndex.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/Migrator.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

os/JournalingObjectStore.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

os/FlatIndex.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

os/FileStore.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

msg/Pipe.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

msg/Pipe.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/OSDMonitor.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/journal.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/events/EMetaBlob.h: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/SnapRealm.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/Migrator.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/MDCache.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/CDir.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

librados/librados.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

kv_flat_btree_async.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

common/Finisher.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge branch 'wip-rbd-import' into next

Reviewed-by: Sage Weil <sage.weil@inktank.com>

mds: Add config option for log segment size

The mds log segment size is chosen from the
default layout object size (4MB).  Add a parameter
to the config to enable setting the log segment
size to an alternate value.

If the config option to set the journal log segment
size is specified, the log layout must be modified
both for the object size and the striping unit size.

Signed-off-by: Sam Lang <sam.lang@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>