RadosModel: RollBack dirties the object

Signed-off-by: Samuel Just <sam.just@inktank.com>

RadosModel: undirty with update_object_undirty

Otherwise it doesn't behave properly with snapshots since
update_object_version doesn't add another snapshot entry.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::trim_object: remove unused int r; declaration

Signed-off-by: Samuel Just <sam.just@inktank.com>

common/hobject: print shard as unsigned, not as char

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::_process_copy_chunk: use append

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: fill in mod_desc for hitset repops

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: use append for write and writefull

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: fill in mod_desc in finish_copyfrom

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::finish_copyfrom: check new_obs.exists, not obs.exists

If the transaction previously created the object, the stat update
will have already happened.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: handle removing the old object in finish_copy_op

do_osd_ops will need to either copy the old version out of the
way or simply delete it depending on mod_desc.  Thus, defer
handling filling that part in until we finish the copy op.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: EOPNOTSUPP on ec_pool if there are omaps in the COPY_GET

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: make COPY_GET async read friendly

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: requeue in_progress_async_reads on_change

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: support async reads on ec pools

Signed-off-by: Samuel Just <sam.just@inktank.com>

src/: add CEPH_OSD_OP_READ_SYNC

EC pools won't support SYNC reads.  This means that TMAP and friends
along with reads from an object class will not be supported.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: use pgbackend->objects_read_sync() as appropriate

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: add sync/async read methods to PGBackend

Signed-off-by: Samuel Just <sam.just@inktank.com>

ObjectStore: _collection_move_rename needs to increment ops

Bugfix.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGLog,PG: reset can_rollback_to when we get a log entry without a transaction

We cannot rollback past an entry which we didn't actually
apply locally.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGLog: we cannot rollback past a recovered object entry

We don't recover the older versions, so we can't do a rollback.
In general, this is ok since we had enough copies to recover it
in the first place.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGLog: create interface allowing interface user to cleanup/rollback

We need to be able to allow the PGLog interface user to provide
logic for rolling back and trimming log entries.  To that end,
serveral PGLog methods now take a LogEntryHander.

In PGLog::merge_old_entry, if prior_version > info.log_tail and
the object is not missing, we must have rolled back the prior
log entry.  Thus, we don't skip the entry.

To simplify the code, _merge_old_entry has been split out as
a const helper.  This way, proc_replica_log can be reexpressed
as merging the divergent replica log entries with the fully
merged authoritative log.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: Fill in rollback info for log events

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: introduce xattr caching for ec pools

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend,ReplicatedBackend: add support for rolling back log entries

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd_types: add a can_rollback_to line to pg_log_t

This allows us to determine how far back the log can be rolled.

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd_types: add ObjectModDesc into pg_log_entry_t

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd_types: ObjectModDesc

Contains information needed to roll back a pg_log_entry_t.

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: move client op handling into ReplicatedBackend

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend: allow queue_transaction to specify an op

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: factor out should_send_op

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: split out op_applied_replica

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG/ReplicatedPG: move snap map management into append_log

Both the replica and the primary should use the same append_log
mechanism to update the snap map.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: implement RPGTransaction

RPGTransaction is essentially a wrapped ObjectStore::Transaction.
The coll_t argument is elided, tempness is instead encoded in the
hobject.  RPGTransaction tracks which temp objects are created and
cleared so we can update the ReplicatedBackend tracking and possibly
create the temp collection as needed.

Signed-off-by: Samuel Just <sam.just@inktank.com>

hobject_t/ReplicatedPG: tempness is now an hobject thing

PGBackend implmentations will have complete control over the temp
collection.  Rather than specifying the collection when sending
ops into the PGBackend, hobjects themselves will be temp or not.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend: add PGTransaction

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: update the collection_list mechanisms to skip stashed objects

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/osd/: verify length of buffer returned in read

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/osd/TestRados.cc: skip 0 weight ops

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/osd: add append support to WriteOp

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/osd: adapt RadosModel to not do omap ops on ec_pools

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/osd: restructure Object/RadosModel in prep for append

Attribute handling no longer has special support in ContentsGenerator.
The most recent operation information is now stored in a special
attr rather than at the beginning of the object.  ObjectDesc layers
include their own ContentsGenerators to allow more flexibility.

Also, writes truncate to the new object size rather than simply
causing reads to stop at that object size.

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/osd/Object: create encoders for ContDesc

Signed-off-by: Samuel Just <sam.just@inktank.com>

src/test/osd: remove tmap test

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGLog: allow for empty pg log in update_range if log_tail == eversion_t()

Signed-off-by: Samuel Just <sam.just@inktank.com>

common: add option to disable filestore sharded object check

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore::lfn_link: handle link within a collection

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: prefer older last_update for ec_pool

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSDMonitor: add debug_fake_ec_pool

This flag will cause ReplicatedPG to act as though the
pool were actually an EC pool in that operations will
be restricted to operations which can be locally rolled
back thereby allowing us to test the ReplicatedPG local
log rollback mechanisms independent of EC.  It will also
cause ReplicatedPG to use the async read mechanism on
the PGBackend implementation once it is implemented.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGLog::rewind_divergent_log: we can rewind to tail, fix assert

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGLog: don't move up log.tail

Moving up log.tail unnecessarily risks backfilling
a replica after a split.  Also, it disrupts the
property that replicas from the most recent interval
which performed writes must have overlapping logs.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: remove pushing entry for each object as it completes

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore::_collection_move_rename: remove source before closing guard

Fixes a bug in _collection_move_rename replays.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend.h: don't need to be active for pushes

Signed-off-by: Samuel Just <sam.just@inktank.com>

ObjectStore: improve name of the queue_transaction which cleans up the transaction

Signed-off-by: Samuel Just <sam.just@inktank.com>

hobject: admit that gen_t is actually version_t

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1079 from ceph/wip-tier-bits

osd: a few tiering fixes

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #1126 from ceph/wip-osdmapenc-fix

MOSDMap: reencode maps if target doesn't have OSDMAP_ENC

Reviewed-by: Sage Weil <sage@inktank.com>

MOSDMap: reencode maps if target doesn't have OSDMAP_ENC

Reencode both full and incremental maps if target doesn't know how to
decode OSDMAP_ENC maps (CEPH_FEATURE_OSDMAP_ENC bit is not set).  This
fixes a compatibility bug that was introduced in 3d7c69fb0986 ("OSDMap:
add a CEPH_FEATURE_OSDMAP_ENC feature, and use new encoding").

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

removing rgw_tests.sh

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

Merge branch 'master' of github.com:ceph/ceph

modified the port to 7280 in the script instead of the default 80

Signed-off-by: tamil <tamil@tamil-VirtualBox.(none)>

Merge pull request #1124 from ceph/wip-cache-mode-fix

mon/MonCommands: 'invalidate+forward' -> 'forward'

Reviewed-by: Sage Weil <sage@inktank.com>

script for rgw bucket quota testing

Signed-off-by: tamil <tamil@tamil-VirtualBox.(none)>

mon/MonCommands: 'invalidate+forward' -> 'forward'

Commit 4e439857a694 introduced invalidate+forward cache mode, commit
81279e3bb6e0 renamed it to forward, but missed the CLI.  Fix it.

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

doc: Fixed end string from '' to `` to stop warnings. Enforced 80 char line.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #1120 from xylv/wip-op-process-latency-squashed

osd: Add op_process_latency in perf counter

Reviewed-by: Sage Weil <sage@inktank.com>

osd: Add op_process_latency in perf counter

Fixes: #7167
Signed-off-by: Ray Lv <xiangyulv@gmail.com>

Merge pull request #1117 from jcsp/wip-7187

Fix #7187: Include all summary items in JSON health output

Backport: emperor, dumpling
Reviewed-by: Sage Weil <sage@inktank.com>

Fix #7187: Include all summary items in JSON health output

Signed-off-by: John Spray <john.spray@inktank.com>

Merge pull request #1112 from kazhang/wip-7180

Fix #7180 in ceph_rest_api

Reviewed-by: Sage Weil <sage@inktank.com>

Missing a key for perm 'w' in permmap (src/pybin/ceph_rest_api.py:277)
It leads to a 500 error when getting mds help info via rest api.
Changed "w" to "rw" in MonCommands.h

Fixes: #7180
Signed-off-by: Kai Zhang <kazhang2@cisco.com>

Merge pull request #1111 from ceph/wip-osdmap-7177

OSDMap: Populate primary_temp values a little more carefully

Reviewed-by: Sage Weil <sage@inktank.com>

OSDMap: Populate primary_temp values a little more carefully

In _get_temp_osds(), we populate temp_pg from the list in the OSDMap,
but we also skip anybody in the list who's down. We need to account
for those skips when setting the primary. It's easy enough to do -- just
look at the output pg_temp list instead of the OSDMap's starting one.

Fixes: #7177
Signed-off-by: Greg Farnum <greg@inktank.com>

fixed the syntax for test maxmds=2

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

fix for the test maxmds=2

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

doc: Fixed ruleset typo.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #1106 from ceph/wip-rgw-contention

Wip rgw contention

Reviewed-by: Sage Weil <sage@inktank.com>

librados: use rwlock for lookup pool, cache results

Use rwlock to read cahed results, only use regular lock if
results are not cached.
Invalidate cache on new osdmap.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add .h file to Makefile rule

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1082 from dachary/wip-ceph-disk

ceph-disk: larger timeout in the test script

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1088 from christian-marie/wip-c-atomic-writes

librados: Add C API coverage for atomic write operations

Reviewed-by: Sage Weil <sage@inktank.com>

doc: Added default-placement to list of placement targets. Added SSL commentary.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added additional comment about port number.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #1092 from ceph/wip-rgw-loadgen

Wip rgw loadgen

Reviewed-by: Sage Weil <sage@inktank.com>

rgw: loadgen, configurable num of objs, buckets

also fix request draining

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: loadgen shutdown, error out on failures

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: sign loadgen requests

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: loadgen frontend read uid, init access key

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add a load generation frontend

Still missing some pieces, but already generates requests.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1107 from xylv/wip-osd-crash-with-leveldb-cache-set

osd: OSD failed to start with osd_leveldb_cache_size > 0

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1109 from ceph/wip-rbd-mount_timeout

rbd: expose mount_timeout map option

Reviewed-by: Sage Weil <sage@inktank.com>

Fixes: #7172
To ensure LevelDBStore::db is destroyed before dependent LevelDBStore::db_cache and LevelDBStore::filterpolicy.

Signed-off-by: Signed-off-by: Ray Lv <xiangyulv@gmail.com>

rbd: expose mount_timeout map option

Expose mount_timeout map option.  (I missed it in commit 9b7364d2450c,
which added -o / --options option and among other options exposed
osdkeepalive and osd_idle_ttl timeouts.)

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

PendingReleaseNotes: note ceph -s fix

See 790dda9c0b75fa5a59ce6187b6131dad54d6d3fe.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1104 from concubidated/master

osd: OSDMap: fix output from ceph status --format=json for num_in_osds

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1097 from ceph/fix-configure-rerunning

avoid re-running configure after initial 'make' invocation

This appears to pass the gitbuilders (and the multiple runs of configure have been irritating me for ages). Yay!

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1103 from ceph/wip-7073

rgw: use configurable prime number for gc hash

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

doc: Removed Calxeda example.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Removed saucy salamander reference.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added host resolution commentary.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>