unittests: fail early when low on disk

Scripts from qa that are run as unittests via test/vstart_wrapper.sh may
fail because the partition on which it runs is low on space ( 95% full
). When it happens the cause of the problem may be unclear because it
is likely to show only as a client not being able to reach the mon.

A test is added in test/vstart_wrapper.sh to verify the disk space usage
using the same method as the mon would and fail with a detailed error if
it is the case.

Signed-off-by: Loic Dachary <loic@dachary.org>

doc: Switched logging to true. Now required for sync agent.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #1102 from ceph/wip-rgw-civetweb-2

Wip rgw civetweb 2

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1116 from jcsp/doc_toofull

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1135 from ceph/use-ceph-sharedptr

libc++: convert tr1 uses

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

libc++: convert tr1 uses

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

doc: Added domain pool, and changed zone configs to use domain tool so they don't clash.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Change zone domain to include "domain" so it is clear.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added space between {Query_String} and [E=HTTP ... to solve for 500 error.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #1134 from ceph/wip-backend-uninit-use

ReplicatedBackend: fix uninitialized use warning/bug

Reviewed-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: fix uninitialized use warning/bug

Signed-off-by: Greg Farnum <greg@inktank.com>

doc: Fixed omission of a comma in zone configuration.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

FileStore: perform LFNIndex lookup without holding fdcache lock

Fixes: #7207
Signed-off-by: Samuel Just <sam.just@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

Merge remote-tracking branch 'origin/next'

mongoose: git rm src/mongoose

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1132 from ceph/wip-erasure-rule

osd/OSDMap: do not create erasure rule by default

Reviewed-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

civetweb: md5.inl -> md5.h

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

civetweb: add include to makefile

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: modify the civetweb rule

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

civetweb: fix module uri

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

civetweb: update submodule

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: switch mongoose to civetweb

Fixes: #7054
mongoose is now GPL, using the civetweb fork instead.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1131 from ceph/use-unordered-map

osd: use ceph:: scoped hash_map

Reviewed-by: Sage Weil <sage@inktank.com>

osd/OSDMap: do not create erasure rule by default

If we do, we will require the v2 feature bit from clients.

We could only include feature bits for rules that are actually referenced
by pools, but for now making the user create the rule is simpler.  There is
no need to create this rule ahead of time.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: use ceph:: scoped hash_map

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #1064 from ceph/port/libc++

support libc++
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge branch 'wip-client-io-final'

Reviewed-by: Greg Farnum <greg@inktank.com>
Reviewed-by: David Zafman <david.zafman@inktank.com>

PGBackend: clarify rollback_stash name and method comments

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: drop messages from down peers

This overlaps with the existing old_peering_msg() mechanism
except in one case: pulls from a replica not in the acting
set.  If such a replica gets marked down, we may resend
pulls to another replica without causing a new interval
to start.  If we recieved, but didn't process, a push in
response to such a pull prior to processing the map marking
the peer down, we might process the push after having reset
the pull state for a different pull operation.  We can
avoid this by discarding ops from down peers.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: make STARTSYNC a noop

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG::calc_acting: consider newest_update_osd when choosing backfill peers

We must include newest_update_osd->second.log_tail when considering backfill
peers because in GetLog we will request logs back to the min last_update over
our acting_backfill set.  This will result in our log being extended as far
backwards as necessary to pick up any peers which can be log recovered by the
union of newest_update_osd's log and that of the chosen primary.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: take ReplicatedPGRef in BlessedContext

Signed-off-by: Samuel Just <sam.just@inktank.com>

RadosModel: RollBack dirties the object

Signed-off-by: Samuel Just <sam.just@inktank.com>

RadosModel: undirty with update_object_undirty

Otherwise it doesn't behave properly with snapshots since
update_object_version doesn't add another snapshot entry.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::trim_object: remove unused int r; declaration

Signed-off-by: Samuel Just <sam.just@inktank.com>

common/hobject: print shard as unsigned, not as char

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::_process_copy_chunk: use append

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: fill in mod_desc for hitset repops

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: use append for write and writefull

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: fill in mod_desc in finish_copyfrom

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::finish_copyfrom: check new_obs.exists, not obs.exists

If the transaction previously created the object, the stat update
will have already happened.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: handle removing the old object in finish_copy_op

do_osd_ops will need to either copy the old version out of the
way or simply delete it depending on mod_desc.  Thus, defer
handling filling that part in until we finish the copy op.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: EOPNOTSUPP on ec_pool if there are omaps in the COPY_GET

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: make COPY_GET async read friendly

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: requeue in_progress_async_reads on_change

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: support async reads on ec pools

Signed-off-by: Samuel Just <sam.just@inktank.com>

src/: add CEPH_OSD_OP_READ_SYNC

EC pools won't support SYNC reads.  This means that TMAP and friends
along with reads from an object class will not be supported.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: use pgbackend->objects_read_sync() as appropriate

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: add sync/async read methods to PGBackend

Signed-off-by: Samuel Just <sam.just@inktank.com>

ObjectStore: _collection_move_rename needs to increment ops

Bugfix.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGLog,PG: reset can_rollback_to when we get a log entry without a transaction

We cannot rollback past an entry which we didn't actually
apply locally.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGLog: we cannot rollback past a recovered object entry

We don't recover the older versions, so we can't do a rollback.
In general, this is ok since we had enough copies to recover it
in the first place.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGLog: create interface allowing interface user to cleanup/rollback

We need to be able to allow the PGLog interface user to provide
logic for rolling back and trimming log entries.  To that end,
serveral PGLog methods now take a LogEntryHander.

In PGLog::merge_old_entry, if prior_version > info.log_tail and
the object is not missing, we must have rolled back the prior
log entry.  Thus, we don't skip the entry.

To simplify the code, _merge_old_entry has been split out as
a const helper.  This way, proc_replica_log can be reexpressed
as merging the divergent replica log entries with the fully
merged authoritative log.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: Fill in rollback info for log events

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: introduce xattr caching for ec pools

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend,ReplicatedBackend: add support for rolling back log entries

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd_types: add a can_rollback_to line to pg_log_t

This allows us to determine how far back the log can be rolled.

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd_types: add ObjectModDesc into pg_log_entry_t

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd_types: ObjectModDesc

Contains information needed to roll back a pg_log_entry_t.

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: move client op handling into ReplicatedBackend

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend: allow queue_transaction to specify an op

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: factor out should_send_op

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: split out op_applied_replica

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG/ReplicatedPG: move snap map management into append_log

Both the replica and the primary should use the same append_log
mechanism to update the snap map.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: implement RPGTransaction

RPGTransaction is essentially a wrapped ObjectStore::Transaction.
The coll_t argument is elided, tempness is instead encoded in the
hobject.  RPGTransaction tracks which temp objects are created and
cleared so we can update the ReplicatedBackend tracking and possibly
create the temp collection as needed.

Signed-off-by: Samuel Just <sam.just@inktank.com>

hobject_t/ReplicatedPG: tempness is now an hobject thing

PGBackend implmentations will have complete control over the temp
collection.  Rather than specifying the collection when sending
ops into the PGBackend, hobjects themselves will be temp or not.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend: add PGTransaction

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: update the collection_list mechanisms to skip stashed objects

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/osd/: verify length of buffer returned in read

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/osd/TestRados.cc: skip 0 weight ops

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/osd: add append support to WriteOp

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/osd: adapt RadosModel to not do omap ops on ec_pools

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/osd: restructure Object/RadosModel in prep for append

Attribute handling no longer has special support in ContentsGenerator.
The most recent operation information is now stored in a special
attr rather than at the beginning of the object.  ObjectDesc layers
include their own ContentsGenerators to allow more flexibility.

Also, writes truncate to the new object size rather than simply
causing reads to stop at that object size.

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/osd/Object: create encoders for ContDesc

Signed-off-by: Samuel Just <sam.just@inktank.com>

src/test/osd: remove tmap test

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGLog: allow for empty pg log in update_range if log_tail == eversion_t()

Signed-off-by: Samuel Just <sam.just@inktank.com>

common: add option to disable filestore sharded object check

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore::lfn_link: handle link within a collection

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: prefer older last_update for ec_pool

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSDMonitor: add debug_fake_ec_pool

This flag will cause ReplicatedPG to act as though the
pool were actually an EC pool in that operations will
be restricted to operations which can be locally rolled
back thereby allowing us to test the ReplicatedPG local
log rollback mechanisms independent of EC.  It will also
cause ReplicatedPG to use the async read mechanism on
the PGBackend implementation once it is implemented.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGLog::rewind_divergent_log: we can rewind to tail, fix assert

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGLog: don't move up log.tail

Moving up log.tail unnecessarily risks backfilling
a replica after a split.  Also, it disrupts the
property that replicas from the most recent interval
which performed writes must have overlapping logs.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: remove pushing entry for each object as it completes

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore::_collection_move_rename: remove source before closing guard

Fixes a bug in _collection_move_rename replays.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend.h: don't need to be active for pushes

Signed-off-by: Samuel Just <sam.just@inktank.com>

ObjectStore: improve name of the queue_transaction which cleans up the transaction

Signed-off-by: Samuel Just <sam.just@inktank.com>

hobject: admit that gen_t is actually version_t

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1079 from ceph/wip-tier-bits

osd: a few tiering fixes

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #1126 from ceph/wip-osdmapenc-fix

MOSDMap: reencode maps if target doesn't have OSDMAP_ENC

Reviewed-by: Sage Weil <sage@inktank.com>

MOSDMap: reencode maps if target doesn't have OSDMAP_ENC

Reencode both full and incremental maps if target doesn't know how to
decode OSDMAP_ENC maps (CEPH_FEATURE_OSDMAP_ENC bit is not set).  This
fixes a compatibility bug that was introduced in 3d7c69fb0986 ("OSDMap:
add a CEPH_FEATURE_OSDMAP_ENC feature, and use new encoding").

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

added perl script for rgw bucket quota tests

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

removing rgw_tests.sh

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>
(cherry picked from commit 54caa0192b02b03549fe4ca5d062495e6e429f97)

removing rgw_tests.sh

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

Merge branch 'master' of github.com:ceph/ceph

modified the port to 7280 in the script instead of the default 80

Signed-off-by: tamil <tamil@tamil-VirtualBox.(none)>

Merge pull request #1124 from ceph/wip-cache-mode-fix

mon/MonCommands: 'invalidate+forward' -> 'forward'

Reviewed-by: Sage Weil <sage@inktank.com>

script for rgw bucket quota testing

Signed-off-by: tamil <tamil@tamil-VirtualBox.(none)>

mon/MonCommands: 'invalidate+forward' -> 'forward'

Commit 4e439857a694 introduced invalidate+forward cache mode, commit
81279e3bb6e0 renamed it to forward, but missed the CLI.  Fix it.

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

Merge pull request #1123 from ceph/wip-stray-mdsmaps

mon/MDSMonitor: do not generate mdsmaps from already-laggy mds
Reviewed-by: Greg Farnum <greg@inktank.com>

doc: Fixed end string from '' to `` to stop warnings. Enforced 80 char line.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>