PGLog: create interface allowing interface user to cleanup/rollback

We need to be able to allow the PGLog interface user to provide
logic for rolling back and trimming log entries.  To that end,
serveral PGLog methods now take a LogEntryHander.

In PGLog::merge_old_entry, if prior_version > info.log_tail and
the object is not missing, we must have rolled back the prior
log entry.  Thus, we don't skip the entry.

To simplify the code, _merge_old_entry has been split out as
a const helper.  This way, proc_replica_log can be reexpressed
as merging the divergent replica log entries with the fully
merged authoritative log.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: Fill in rollback info for log events

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: introduce xattr caching for ec pools

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend,ReplicatedBackend: add support for rolling back log entries

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd_types: add a can_rollback_to line to pg_log_t

This allows us to determine how far back the log can be rolled.

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd_types: add ObjectModDesc into pg_log_entry_t

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd_types: ObjectModDesc

Contains information needed to roll back a pg_log_entry_t.

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: move client op handling into ReplicatedBackend

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend: allow queue_transaction to specify an op

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: factor out should_send_op

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: split out op_applied_replica

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG/ReplicatedPG: move snap map management into append_log

Both the replica and the primary should use the same append_log
mechanism to update the snap map.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: implement RPGTransaction

RPGTransaction is essentially a wrapped ObjectStore::Transaction.
The coll_t argument is elided, tempness is instead encoded in the
hobject.  RPGTransaction tracks which temp objects are created and
cleared so we can update the ReplicatedBackend tracking and possibly
create the temp collection as needed.

Signed-off-by: Samuel Just <sam.just@inktank.com>

hobject_t/ReplicatedPG: tempness is now an hobject thing

PGBackend implmentations will have complete control over the temp
collection.  Rather than specifying the collection when sending
ops into the PGBackend, hobjects themselves will be temp or not.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend: add PGTransaction

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: update the collection_list mechanisms to skip stashed objects

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/osd/: verify length of buffer returned in read

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/osd/TestRados.cc: skip 0 weight ops

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/osd: add append support to WriteOp

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/osd: adapt RadosModel to not do omap ops on ec_pools

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/osd: restructure Object/RadosModel in prep for append

Attribute handling no longer has special support in ContentsGenerator.
The most recent operation information is now stored in a special
attr rather than at the beginning of the object.  ObjectDesc layers
include their own ContentsGenerators to allow more flexibility.

Also, writes truncate to the new object size rather than simply
causing reads to stop at that object size.

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/osd/Object: create encoders for ContDesc

Signed-off-by: Samuel Just <sam.just@inktank.com>

src/test/osd: remove tmap test

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGLog: allow for empty pg log in update_range if log_tail == eversion_t()

Signed-off-by: Samuel Just <sam.just@inktank.com>

common: add option to disable filestore sharded object check

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore::lfn_link: handle link within a collection

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: prefer older last_update for ec_pool

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSDMonitor: add debug_fake_ec_pool

This flag will cause ReplicatedPG to act as though the
pool were actually an EC pool in that operations will
be restricted to operations which can be locally rolled
back thereby allowing us to test the ReplicatedPG local
log rollback mechanisms independent of EC.  It will also
cause ReplicatedPG to use the async read mechanism on
the PGBackend implementation once it is implemented.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGLog::rewind_divergent_log: we can rewind to tail, fix assert

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGLog: don't move up log.tail

Moving up log.tail unnecessarily risks backfilling
a replica after a split.  Also, it disrupts the
property that replicas from the most recent interval
which performed writes must have overlapping logs.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: remove pushing entry for each object as it completes

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore::_collection_move_rename: remove source before closing guard

Fixes a bug in _collection_move_rename replays.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend.h: don't need to be active for pushes

Signed-off-by: Samuel Just <sam.just@inktank.com>

ObjectStore: improve name of the queue_transaction which cleans up the transaction

Signed-off-by: Samuel Just <sam.just@inktank.com>

hobject: admit that gen_t is actually version_t

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1079 from ceph/wip-tier-bits

osd: a few tiering fixes

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #1126 from ceph/wip-osdmapenc-fix

MOSDMap: reencode maps if target doesn't have OSDMAP_ENC

Reviewed-by: Sage Weil <sage@inktank.com>

MOSDMap: reencode maps if target doesn't have OSDMAP_ENC

Reencode both full and incremental maps if target doesn't know how to
decode OSDMAP_ENC maps (CEPH_FEATURE_OSDMAP_ENC bit is not set).  This
fixes a compatibility bug that was introduced in 3d7c69fb0986 ("OSDMap:
add a CEPH_FEATURE_OSDMAP_ENC feature, and use new encoding").

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

removing rgw_tests.sh

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

Merge branch 'master' of github.com:ceph/ceph

modified the port to 7280 in the script instead of the default 80

Signed-off-by: tamil <tamil@tamil-VirtualBox.(none)>

Merge pull request #1124 from ceph/wip-cache-mode-fix

mon/MonCommands: 'invalidate+forward' -> 'forward'

Reviewed-by: Sage Weil <sage@inktank.com>

script for rgw bucket quota testing

Signed-off-by: tamil <tamil@tamil-VirtualBox.(none)>

mon/MonCommands: 'invalidate+forward' -> 'forward'

Commit 4e439857a694 introduced invalidate+forward cache mode, commit
81279e3bb6e0 renamed it to forward, but missed the CLI.  Fix it.

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

doc: Fixed end string from '' to `` to stop warnings. Enforced 80 char line.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #1120 from xylv/wip-op-process-latency-squashed

osd: Add op_process_latency in perf counter

Reviewed-by: Sage Weil <sage@inktank.com>

osd: Add op_process_latency in perf counter

Fixes: #7167
Signed-off-by: Ray Lv <xiangyulv@gmail.com>

Merge pull request #1117 from jcsp/wip-7187

Fix #7187: Include all summary items in JSON health output

Backport: emperor, dumpling
Reviewed-by: Sage Weil <sage@inktank.com>

Fix #7187: Include all summary items in JSON health output

Signed-off-by: John Spray <john.spray@inktank.com>

Merge pull request #1112 from kazhang/wip-7180

Fix #7180 in ceph_rest_api

Reviewed-by: Sage Weil <sage@inktank.com>

Missing a key for perm 'w' in permmap (src/pybin/ceph_rest_api.py:277)
It leads to a 500 error when getting mds help info via rest api.
Changed "w" to "rw" in MonCommands.h

Fixes: #7180
Signed-off-by: Kai Zhang <kazhang2@cisco.com>

Merge pull request #1111 from ceph/wip-osdmap-7177

OSDMap: Populate primary_temp values a little more carefully

Reviewed-by: Sage Weil <sage@inktank.com>

OSDMap: Populate primary_temp values a little more carefully

In _get_temp_osds(), we populate temp_pg from the list in the OSDMap,
but we also skip anybody in the list who's down. We need to account
for those skips when setting the primary. It's easy enough to do -- just
look at the output pg_temp list instead of the OSDMap's starting one.

Fixes: #7177
Signed-off-by: Greg Farnum <greg@inktank.com>

fixed the syntax for test maxmds=2

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

fix for the test maxmds=2

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

doc: Fixed ruleset typo.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #1106 from ceph/wip-rgw-contention

Wip rgw contention

Reviewed-by: Sage Weil <sage@inktank.com>

librados: use rwlock for lookup pool, cache results

Use rwlock to read cahed results, only use regular lock if
results are not cached.
Invalidate cache on new osdmap.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add .h file to Makefile rule

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1082 from dachary/wip-ceph-disk

ceph-disk: larger timeout in the test script

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1088 from christian-marie/wip-c-atomic-writes

librados: Add C API coverage for atomic write operations

Reviewed-by: Sage Weil <sage@inktank.com>

doc: Added default-placement to list of placement targets. Added SSL commentary.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added additional comment about port number.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #1092 from ceph/wip-rgw-loadgen

Wip rgw loadgen

Reviewed-by: Sage Weil <sage@inktank.com>

rgw: loadgen, configurable num of objs, buckets

also fix request draining

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: loadgen shutdown, error out on failures

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: sign loadgen requests

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: loadgen frontend read uid, init access key

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add a load generation frontend

Still missing some pieces, but already generates requests.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1107 from xylv/wip-osd-crash-with-leveldb-cache-set

osd: OSD failed to start with osd_leveldb_cache_size > 0

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1109 from ceph/wip-rbd-mount_timeout

rbd: expose mount_timeout map option

Reviewed-by: Sage Weil <sage@inktank.com>

Fixes: #7172
To ensure LevelDBStore::db is destroyed before dependent LevelDBStore::db_cache and LevelDBStore::filterpolicy.

Signed-off-by: Signed-off-by: Ray Lv <xiangyulv@gmail.com>

rbd: expose mount_timeout map option

Expose mount_timeout map option.  (I missed it in commit 9b7364d2450c,
which added -o / --options option and among other options exposed
osdkeepalive and osd_idle_ttl timeouts.)

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

PendingReleaseNotes: note ceph -s fix

See 790dda9c0b75fa5a59ce6187b6131dad54d6d3fe.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1104 from concubidated/master

osd: OSDMap: fix output from ceph status --format=json for num_in_osds

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1097 from ceph/fix-configure-rerunning

avoid re-running configure after initial 'make' invocation

This appears to pass the gitbuilders (and the multiple runs of configure have been irritating me for ages). Yay!

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1103 from ceph/wip-7073

rgw: use configurable prime number for gc hash

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

doc: Removed Calxeda example.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Removed saucy salamander reference.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added host resolution commentary.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added comments from #6682.

fixes: #6682

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added modification to quick start.

fixes: #6682

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Changed file name.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added source file for graphic.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added source file for graphic.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added third monitor icon.

fixes: #6959

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added additional monitor icon.

fixes: #6959

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Removed old file. Changed file name.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

rgw: use rwlock for cache

Use a window for cache lru updates, so that we don't need to get the
writer lock every read.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

osd: OSDMap: fix output from ceph status --format=json for num_in_osds
num_up_osds returns as an int value, while num_in_osds returns as a string.
Since only an int can be returned from get_num_in_osds(), num_in_osds should
should also be an int to remain consistant with num_up_osds.

Fixes: 7159
Signed-off-by: Tyler Brekke <tyler.brekke@inktank.com>

Merge pull request #915 from ceph/wip-rgw-expose-bucket

6748: rgw: Optionally return the bucket name in a response header.

Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

doc: Fixed keyring command and updated for current conventions.

fixes: #6692

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #1094 from ceph/wip-7117-redhat

packaging: ship libdir/ceph

Reviewed-by: Sage Weil <sage@inktank.com>

mon: larger timeout for mon mkfs.sh test

Otherwise slow machines may timeout just because they are slow and not
because the process is blocked by an actual problem.

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-disk: larger timeout in the test script

Otherwise slow machines may timeout just because they are slow and not
because the process is blocked by an actual problem.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1095 from ceph/wip-7166

osd: OSDMap: build reverse name->pool map upon decoding

Reviewed-by: Loic Dachary <loic@dachary.org>

osd: OSDMap: build reverse name->pool map upon decoding

Commit 3d7c69fb09 introduced a new OSDMap encoding/decoding scheme.
However, while the classic decoding function still kept building the
reverse name->pool map, the new decoding function did not, causing the
monitor to be unable to map pool names to pool ids.

This patch fixes this, by factoring out the loop responsible for
populating the 'name_pool' map, as well as calling 'calc_num_osds()', to
OSDMap::post_decode() and having this function called from both the
classic and the new decode functions.

Fixes: 7166
Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>

6748: rgw: Optionally return the bucket name in a response header.

This can be useful in situations where accounting of traffic is done externally
when for example HTTP traffic is cached by a reverse proxy like Varnish.

Since not all traffic reaches the RGW daemon it can't fully account all traffic
and this the caching proxy needs to be aware of which bucket the request came for.

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #1086 from ceph/wip-temp-primary

Add a "primary temp" to go with "pg temp"

Reviewed-by: Sage Weil <sage@inktank.com>

packaging: ship libdir/ceph

Automake puts ceph_common.sh into libdir/ceph, but the Red Hat packaging
was not capturing this file.

Add the libdir/ceph location to the RPM packaging.

Fixes #7117