librbd: invalidate cache when flattening

The cache stores which objects don't exist. Flatten bypasses the cache
when doing its copyups, so when it is done the -ENOENT from the cache
is treated as zeroes instead of 'need to read from parent'.

Clients that have the image open need to forgot about the cached
non-existent objects as well. Do this during ictx_refresh, while the
parent_lock is held exclusively so no new reads from the parent can
happen until the updated parent metadata is visible, so no new reads
from the parent will occur.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

ObjectCacher: add a method to clear -ENOENT caching

Clear the exists and complete flags for any objects that have exists
set to false, and force any in-flight reads to retry if they get
-ENOENT instead of generating zeros.

This is useful for getting the cache into a consistent state for rbd
after an image has been flattened, since many objects which previously
did not exist and went up to the parent to retrieve data may now exist
in the child.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

ObjectCacher: keep track of outstanding reads on an object

Reads always use C_ReadFinish as a callback (and they are the only
user of this callback). Keep an xlist of these for each object, so
they can remove themselves as they finish. To prevent racing requests
and with discard removing objects from the cache, clear the xlist in
the object destructor, so if the Object is still valid the set_item
will still be on the list.

Make the ObjectCacher constructor take an Object* instead of the pool
and object id, which are derived from the Object* anyway.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

config: note which options are overridden by common_preinit()

Defaults for these differ based on the context in which they're used.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

common: reduce default in-memory logs for non-daemons

The default of 100000 can result in hundreds of MBs of extra memory
used. This was most obvious when using librbd with caching enabled,
since there was a dout(0) accidentally left in the ObjectCacher.

refs: #4352
backport: bobtail
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

ObjectCacher: fix debug log level in split

Level 0 should never be used for this kind of debugging.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>

Merge pull request #90 from grosskur/fix-debian-libsnappy

debian: require libsnappy-dev for ceph

debian: require libsnappy-dev for ceph

Debian builds are currently broken without this requirement.

release-process.rst:  Fix typos

Signed-off-by: Gary Lowell <gary.lowell@inktank.com>

Merge branch 'wip-prepare'

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Reviewed-by: Alexandre Marangone <alexandre.marangone@inktank.com>
Tested-by: Tamil Muthamizhan <tamil.muthamizhan@inktank.com>

ceph-disk-prepare: move in-use checks to the top, before zap

Move the in-use checks to the very top, before we (say) zap!

Signed-off-by: Sage Weil <sage@inktank.com>

doc/release-notes: v0.58

Signed-off-by: Sage Weil <sage@inktank.com>

Update doc/radosgw/s3/python.rst

Note about creating conn object for non-ssl implementations.

Merge branch 'wip-4207'

Feature: #4207: osd/librados: add ops to list snaps for an object

Signed-off-by: David Zafman <david.zafman@inktank.com>
Reported-by: Sam Just <sam.just@inktank.com>

Merge branch 'master' of https://github.com/ceph/ceph

Merge branch 'next'

Add list_snaps() calls to snapshots test cases

Signed-off-by: David Zafman <david.zafman@inktank.com>

Add rados listsnaps command

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd/librados: add op to list clones/snaps for an object

Returning snap_set_t with clone info
and snapshots in ascending order
Add clones with snapshots to obj_list_snap_response_t
New rados_types.hpp with snap_set_t/clone_info_t
Move snap_t to rados_types.hpp
Add generate_test_instances() and TYPE() to encoding/types.h

Feature: #4207

Signed-off-by: David Zafman <david.zafman@inktank.com>

Remove unused SnapContext in librados.hpp

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd: Improve snapshot test

In SelfManagedSnapRollbackPP add some overlapping writes
New SelfManagedSnapOverlap creates overlapping writes

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd: correct comment

SnapSet snaps are in descending order

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph-disk-prepare: verify device is not in use by device-mapper

Be nice and tell the user which devices/mappings are consuming the device,
too.

Signed-off-by: Sage Weil <sage@inktank.com>

ceph-disk-prepare: verify device is not mounted before using

Make sure the data and/or journal device(s) are not in use (mounted)
before using them.  Make room for additional "in-use" checks in the future.

Closes: #3256
Signed-off-by: Sage Weil <sage@inktank.com>

debian: require >= python2.6 for ceph as well

ceph-disk-prepare and ceph-disk-activate use the with statement,
str.format, and possibly other new features from python 2.6.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

ceph-disk-prepare: clean up stupid check for a digit

Signed-off-by: Sage Weil <sage@inktank.com>

ceph-disk-prepare: use os.path.realpath()

My janky symlink resolution is broken in various ways.

Signed-off-by: Sage Weil <sage@inktank.com>

v0.58

Add X-Python-Version >=2.6 to debian control file.

python-ceph complains when installed to debian squeeze about the 'with'
statement. Apparently installation tries to install the python-ceph package for
python 2.5, which does not support the 'with' statement natively.

Signed-off-by: Filippos Giannakos <philipgian@grnet.gr>

PG,ReplicatedPG: use pg_has_reset_since to discard old async events

Local async events are obsolete if the pg is deleting or if the
epoch at which they were created is prior to last_peering_reset.

Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 62999ea2d362b32dfc86a15dcb03b06fb5cb0572)

PG::build_scrub_map: detect race with peering via last_peering_reset

Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 67225339dc3d62d7fe5a32eec65d51e53e8d35bb)

ReplicatedPG::C_OSD_CommittedPushedObject: use intrusive_ptr for pg

Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 04ee8f478bbd587a711d0668c471cfc5c1cab06c)

ReplicatedPG::C_OSD_CommittedPushedObject take epoch submitted

What we really care about is that the epoch in which the Context
was submitted is at complete() time >= last_peering_reset.

Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit a01dea6af9aacf0614570ebb5fa161d9dde9b6b6)

init-ceph: fix --restart option

Reported-by: Bryan K. Wright <bkw1a@ayesha.phys.virginia.edu>
Signed-off-by: Sage Weil <sage@inktank.com>

doc: Updated year in footer.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge branch 'master' of https://github.com/ceph/ceph

doc: Added instructions for ensuring ceph config file and keyring prior to mounting ceph fuse.

fixes: #3796

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #89 from Elbandi/master

remove unused variable from makefile

jounal: disable aio

There is a deadlock issue in the aio code, see #4079.  Disable for the time
being.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/wip-3484'

Reviewed-by: Sage Weil <sage@inktank.com>

Build: no longer need to use LEVELDB_INCLUDE

Signed-off-by: Andras Elso <elso.andras@gmail.com>

tools: ceph: add 'ceph df' usage

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: Monitor: 'ceph df'

Fixes: #3484
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

README:  Add libsnappy-dev to build requirements.

Signed-off-by: Gary Lowell <gary.lowell@inktank.com>

doc: missing blank line in radosgw/config-ref

Merge pull request #85 from dalgaaf/wip-da-spec-1

Fix package requirements to get sgdisk and merge some changes back from SUSE spec file.

Reviewed-by: Gary Lowell <gary.lowell@inktank.com>

Merge pull request #88 from nmtadam/master

librados.hpp typo

librados.hpp: replace paris with pairs

Signed-off-by: Adam Manzanares <nmtadam@gmail.com>
Working on a lua-rados library I noticed that paris was used
in the place of pairs in some comments.

doc: Instruction should have been noout, not nodown.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Fixed OSD pool config settings. Should be in [global], not [osd].

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

ceph.spec.in: Require gptfdisk for Suse, gdisk for otherwise

Signed-off-by: Gary Lowell <gary.lowell@inktank.com>

Merge pull request #87 from dalgaaf/wip-da-sca-style-2

Fix more issues found via cppcheck

Reviewed-by: Sage Weil <sage@inktank.com>

osd/Watch.cc: init NotifyTimeoutCB::canceled in constructor with false

Initialize NotifyTimeoutCB::canceled in the constructor with
false as already done in onter cases.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/ReplicatedPG.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ReplicatedPG.cc: remove unused variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ReplicatedPG.cc: remove consecutive return/break statement

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/PG.cc: reduce scope of listed_collection in read_log_old()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/PG.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/OSD.cc: remove unused variable 'map<int64_t, int> pool_resize'

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/OSD.cc: reduce scope of "PG *pg" in OSD::handle_pg_query()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/OSD.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

os/IndexManager.cc: reduce scope of 'int r' in IndexManager::build_index()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

os/LevelDBStore.h: pass const string by reference to past_prefix()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

os/FileStore.cc: fix warning about comparing int with bool

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

os/FileStore.cc: remove identical else branch

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

os/FileStore.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

SimpleMessenger.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

msg/Pipe.cc: reduce scope of some variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/Paxos.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/PGMonitor.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/PGMap.cc: remove useless assigment to 'first', it's never used

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/OSDMonitor.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/MonmapMonitor.cc: remove unused variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/MonmapMonitor.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/Monitor.cc: reduce scope of reuse_caps in Monitor::_ms_dispatch()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/Monitor.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/MonClient.cc: use static_cast instead of C-Style cast

Use static_cast<T*>(m) instead of C-Style cast, add missing
include needed for static_cast.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/MDSMonitor.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/LogMonitor.cc: move unused variable into related code block

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/LogMonitor.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/Elector.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/AuthMonitor.cc: remove unused 'std::stringstream ss'

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/flock.cc: remove unused variable 'crossed_waiting_locks'

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/Server.cc: reduce scope of some variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/Server.cc: : use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/Migrator.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/MDS.cc: remove unused variable 'oldresolve'

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/MDS.cc: : use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/MDCache.cc: remove unused variables recover_q and check_q

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/MDCache.cc: reduce scope of 'in' in MDCache::open_root_inode()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/MDCache.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/MDBalancer.h: init some double variables in constructor

Initialize double variables my_load and target_load with 0.0 in
the constructor.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/MDBalancer.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ESession.h: initialze 'open' in the constructor with false

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/LogEvent.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/Locker.cc: reduce scope of 'pool' in Locker::issue_client_lease()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph.spec.in: merge fdupes support from SUSE packages

Use fdupes to avoid duplicated files as done in the SUSE
packages. Since it doesn't work as expected on Fedora/RHEL
do this only on SUSE based distros.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph.spec.in: add on non-RHEL sharutils to 'BuildRequires:'

Add sharutils to 'BuildRequires:', needed for configure.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph.spec.in: fix 'Requires:' to get sgdisk into the system

Fix package Requirements to get sgdisk. On RHEL/CentOS/Fedora the
package is gdisk. On SUSE/openSUSE < 12.2 it's scsirastools on
openSUSE >= 12.2 it's in gptfdisk.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/Locker.cc: use static_cast instead of C-Style cast

Use static_cast<T>() instead of C-Style cast.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>