OSD,config_opts: log osd state changes at level 0 instead

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd: be a bit more verbose on startup

load_pgs can take a while and it is nice to know what ceph-osd is doing
without cranking up logging.

Did a quick audit of dout(1)'s and making this the default.  This lets us
see basic OSD state changes (load_pgs, boot, active) at the default level.

At this point all osd state changes should be logged at level 1.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1331 from ceph/wip-cache-pool

mon/OSDMonitor: make default false-positive-probability 5%

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #1332 from ceph/wip-pg-msg

mon/OSDMonitor: missing space in string

Reviewed-by: Sage Weil <sage@inktank.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

mon/OSDMonitor: missing space in string

Minor glitch.  Was printing ..."exceeds per-OSD max of32)"

Signed-off-by: John Spray <john.spray@inktank.com>

Fix python-requests package dependencies.

python-ceph does not require requests, but ceph-common does (for ceph-brag).

Signed-off-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit 9a0ef6a181e90b81cdccbd54298270f5aa960767)

mon/OSDMonitor: make default false-positive-probability 5%

This is a more conservative default (as in, less memory consumed) for
newly created cache pools.

Signed-off-by: Sage Weil <sage@inktank.com>

mds: fix nested_anchors update during journal replay

check if the inode is anchored/unanchored before updating the inode

Fixes: #7530
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #1319 from ceph/wip-primary-temp-fix

osd/OSDMap: respect temp primary without temp acting

Reviewed-by: Greg Farnum <greg@inktank.com>

osd/OSDMap: respect temp primary without temp acting

be2748c6d540891f2e1a62e7034cb44f7d04bf18 ensured that
if the temp acting mapping contains only CRUSH_ITEM_NONE,
that the acting_primary is left at -1.  However, even if
acting.empty(), we need to respect a temp_primary mapping.
Thus, use _acting_primary unless acting.empty() &&
acting_primary == -1.

Bug introduced in be2748c6d540891f2e1a62e7034cb44f7d04bf18.
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1317 from ceph/wip-7537

Wip 7537

Reviewed-by: David Zafman <david.zafman@inktank.com>

OSDMap::_pg_to_up_acting_osds: use _acting_primary unless acting is empty

If the temp set for whatever reason has only CRUSH_ITEM_NONE,
we need primary to be -1.

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1311 from ceph/wip-dz-scrub-fixes

Wip dz scrub fixes

Reviewed-by: Samuel Just <sam.just@inktank.com>

OSDMonitor: when thrashing, only generate valid temp pg mappings

Since backfill peers are no longer placed into the acting set,
temp mappings will never exceed the pool size.  Also, for ec
pools, temp mappings will never be less than the pool size.

Signed-off-by: Samuel Just <sam.just@inktank.com>

Revert "osd/PG: fix assert when deep repair finds no errors"

This reverts commit e3e3328ec8a57f1ae2a2fd9893c51068798720a0.

osd: Don't include primary's shard in repair result message

Signed-off-by: David Zafman <david.zafman@inktank.com>

Merge pull request #1308 from ceph/wip-osdmap-inc

mon/OSDMonitor: fix osdmap encode feature logic

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #1302 from ceph/wip-create-null

client: fix possible null dereference in create

Reviewed-by: Greg Farnum <greg@inktank.com>

ceph_test_objectstore: fix i386 build (again)

test/objectstore/store_test.cc: In member function ‘void SyntheticWorkloadState::read()’:
error: test/objectstore/store_test.cc:462:23: no matching function for call to ‘swap(uint64_t&, size_t&)’

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1307 from ceph/wip-7517

Wip 7517

mon/OSDMonitor: fix osdmap encode feature logic

If we are encoding a full map based on an old Incremental that does not
encode the features, fall back to the quorum features or (barring that)
all features.  Do *not* do no features or else we will end up with
encode_client_old which does not even include the extended info and will
cause the mon to crash when decoding.

This was observed when upgading a 0.76 cluster to 0.77 (all mons stopped,
upgraded, and then started)

Reported-by: Aaron Ten Clay <aarontc@aarontc.com>
Signed-off-by: Sage Weil <sage@inktank.com>

PG: skip pg_whoami.osd, not pg_whoami.shard in scrub feature check

Caused by typo in 68184d4574cd507ab3a065693e392917b98e16ec.

Fixes: #7517
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1304 from ceph/wip-last-epoch-clean

mon: fix min_last_epoch_clean handling

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1306 from ceph/wip-7512

wip-7512

Reviewed-by: Sage Weil <sage@inktank.com>

OSD::project_pg_history needs to account for acting_primary/up_primary

Fixes: #7512
Signed-off-by: Samuel Just <sam.just@inktank.com>

Objecter/OSDMap: factor out primary_changed() into static OSDMap method

We need to reuse this logic in OSD::project_pg_history.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: clarify same_primary_since updates regarding primary rank

pg_shard_t includes the position, so these checks handle
same osd/different rank properly.

Signed-off-by: Samuel Just <sam.just@inktank.com>

mon/PGMap: fix osd_epochs update

The insert() call here does not overwrite a previous entry, which means
that the osd_epochs map is never moving forward in time.  This seems to
have been broken since it was introduced in 091809b814.

Backport: emperor, dumpling
Signed-off-by: Sage Weil <sage@inktank.com>

mon/PGMap: add unit test for min_last_epoch_clean

Signed-off-by: Sage Weil <sage@inktank.com>

ECBackend: skip canceled xattr reads as well

Signed-off-by: Samuel Just <sam.just@inktank.com>

client: fix possible null dereference in create

There are two paths that jump to the out label for which 'in' can be
NULL and outp can be non-NULL.  For those cases we want to fill in the
caller's pointer value (they asked for it) but we clearly cannot take
a reference.

Backport: emperor, dumpling
Signed-off-by: Sage Weil <sage@inktank.com>

osd: increase default leveldb write buffer, cache size

The FileStore's leveldb currently uses libleveldb's defaults for cache and
write buffer size, which are both 4 MB. Increase the cache size to 128MB and
the write buffer to 8MB.

Tested-by: Dmitry Smirnov <onlyjob@member.fsf.org>
Signed-off-by: Sage Weil <sage@inktank.com>

ceph_test_objectstore: fix i386 build error

We can't swap size_t and uint64_t; just change the len type to match.

Signed-off-by: Sage Weil <sage@inktank.com>

ceph_test_objectstore: fix signed/unsigned warning

In file included from test/objectstore/store_test.cc:33:0:
../src/gtest/include/gtest/gtest.h: In function ‘testing::AssertionResult testing::internal::CmpHelperEQ(const char*, const char*, const T1&, const T2&) [with T1 = int, T2 = unsigned int]’:
../src/gtest/include/gtest/gtest.h:1300:30: instantiated from ‘static testing::AssertionResult testing::internal::EqHelper::Compare(const char*, const char*, const T1&, const T2&) [with T1 = int, T2 = unsigned int, bool lhs_is_null_literal = false]’
test/objectstore/store_test.cc:484:193: instantiated from here
warning: ../src/gtest/include/gtest/gtest.h:1263:3: comparison between signed and unsigned integer expressions [-Wsign-compare]

Signed-off-by: Sage Weil <sage@inktank.com>

erasure-code: test that changing the pool size is not allowed

http://tracker.ceph.com/issues/7360 fixes #7360

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1294 from ceph/wip-librados-object-iterator

fix bugs in librados::ObjectIterator

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1295 from yuyuyu101/keyvaluestore-enhance

Keyvaluestore enhance(backport to firely)

Pulling this into firefly because it doesn't (substantiatively) touch anything outside of KeyValueStore.

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1297 from ceph/wip-ec-bugs

Wip ec bugs

Reviewed-by: Sage Weil <sage@inktank.com>

unittest_ecbackend: fix signed/unsigned warnings

In file included from test/osd/TestECBackend.cc:20:0:
../src/gtest/include/gtest/gtest.h: In instantiation of ‘testing::AssertionResult testing::internal::CmpHelperEQ(const char*, const char*, const T1&, const T2&) [with T1 = long unsigned int; T2 = int]’:
../src/gtest/include/gtest/gtest.h:1300:30: required from ‘static testing::AssertionResult testing::internal::EqHelper::Compare(const char*, const char*, const T1&, const T2&) [with T1 = long unsigned int; T2 = int; bool lhs_is_null_literal = false]’
test/osd/TestECBackend.cc:30:281: required from here
warning: ../src/gtest/include/gtest/gtest.h:1263:3: comparison between signed and unsigned integer expressions [-Wsign-compare]

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1301 from ceph/wip-scrub-primary

mon/PGMonitor: fix primary osd check on deep-scrub

gitbuilders are now passing

mon/PGMonitor: fix primary osd check on deep-scrub

s/!=/==/.  Logic was reversed.

Broken in 40bdcb88504aea6288d461d29d24d5b0bf7aeebc.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1300 from ceph/wip-7513

PGLog::activate_not_complete typo

Reviewed-by: David Zafman <david.zafman@inktank.com>

PGLog::activate_not_complete: fix log.complete_to increment typo

info.last_complete should be the entry before log.complete_to.

This appears to have been a typo introduced in
dd71051a8f6ec611b3559f8e2aff3c001be37792.

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1298 from ceph/wip-subscribe

osd: fix off-by-one is boot subscription

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1299 from ceph/wip-repair

osd/PG: fix assert when a shallow repair has fixed all errors.

Reviewed-by: David Zafman <david.zafman@inktank.com>

osd/PG: fix assert when deep repair finds no errors

If there are no deep repairs, we don't want to assert.
Fixes:

    -1> 2014-02-21 21:13:56.393087 7f0258ff9700  0 log [INF] : 0.0 repair ok, 0 fixed
     0> 2014-02-21 21:13:56.428703 7f0258ff9700 -1 osd/PG.cc: In function 'void PG::scrub_finish()' thread 7f0258ff9700 time 2014-02-21 21:13:56.393127
osd/PG.cc: 4294: FAILED assert(deep_scrub)

Signed-off-by: David Zafman <david.zafman@inktank.com>
Signed-off-by: Sage Weil <sage@inktank.com>

PGLog: test for missing using the correct shard value

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: fix query for ec pgs

We need to parse a pg_t and translate it into an spg_t.

Signed-off-by: Samuel Just <sam.just@inktank.com>

Objecter: make is_pg_changed static and const for old/newacting

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSDMap: make calc_pg_rank/role const for acting

Signed-off-by: Samuel Just <sam.just@inktank.com>

Objecter: resend op if primary role changed

Fixes: #7508
Signed-off-by: Samuel Just <sam.just@inktank.com>

ECTransaction: require hash_infos for deleted objects

Otherwise, an append preceded by a delete may not see the
correct HashInfo state since it won't have been cached
until the delete is applied.

Fixes: #7506
Signed-off-by: Samuel Just <sam.just@inktank.com>

ECUtil: clear() should reset hashes to -1, not 0

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::cancel_pull: ECBackend might cancel_pull on a non-missing object

Signed-off-by: Samuel Just <sam.just@inktank.com>

ECBackend::filter_read_op: clean up read state properly

Fixes: #7494
Signed-off-by: Samuel Just <sam.just@inktank.com>

osd: fix off-by-one is boot subscription

If we have osdmap N, we want to onetime subscribe
starting at N+1.  Among other things, it means we
hear when the NOUP flag is cleared.

This appears to have broken somewhere around
3c76b81f2f96b790b72f2088164ed8e9d5efbba1.

Fixes: #7511
Signed-off-by: Sage Weil <sage@inktank.com>

Rename keyvaluestore_check_backend to keyvaluestore_debug_check_backend

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Add KeyValueStore op thread options

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Remove eio inject codes in KeyValueStore

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Add config tracker to KeyValueStore

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Add backend check option to KeyValueStore

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Remove filestore_inject_stall

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Remove m_eio in KeyValueStore

KeyValueStore use kv backend to detect errors and nearly can't know the actual
reason for error.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Make SequencePosition sync by each transaction

Each object modify will increase SequencePosition and sync it to disk

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Implement collection_rename interface in KeyValueStore

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Remove unnecessary "check_coll" check

The goal of "check_coll" is aimed to ensure the collection is exists. But
the create and delete of collection already ensured by OSD, just remove check.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Optimize write call add enhance error detect

A single write call may need several keys in the backend kv store. Let get these
keys one time.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Unify object level lock in GenericObjectMap

Before we copy lock implementation from DBObjectMap which provide with two locks
for header. Here just unify it to make ease.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Fix incorrect read and truncate

If the first chunk of object is not read entirely, the current implementation
will contain it entirely which make wrong.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Return 0 when the offset of read exceed the length of object

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Add read/write operation to store_test.cc

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Adjust some interfaces to optimize caller codes

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Use rename operation to make colection_move atomic

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Add get_*_with_header to StripObjectHeader

In some situations, we need to ensure header is held and try to read, so read
interfaces need to accept header argument to allow.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Add test for omap interface

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Move perf counter and add op queue reserve throttle

The perf counter of FileStore can shared with other ObjectStore backend, so move
it to ObjectStore and adjust position to let other KeyValueStore refer to.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Fix deadlock caused by hold collection obj

Collection is a special object in KeyValueStore, if exists collection modify,
this object will be hold. When exists check collection, it will be dead lock to
try to access this object.

Now lookup transaction cache first to ensure no collection object exists
already.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge branch 'wip-6685-firefly' into firefly

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1293 from jwriteclub/doc-update-ceph-osd-note

Added a note that the ceph-osd command requires the cluster option.

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Added a note that the ceph-osd command requires the cluster option.

Signed-off-by: Christopher O'Connell <jwriteclub@gmail.com>

librados: implement ObjectIterator copying and assignment

Copied iterators should be independent of each other, and not
rely on the same underlying ListContext or ObjListCtx.

Add some tests for this to make sure they work in various
circumstances.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

Merge remote-tracking branch 'origin/wip-da-SCA-20140218' into firefly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Added evaluation of cluster and fixed evaluation when setting initial weight on start in verbose mode

Signed-off-by: Christopher O'Connell <jwriteclub@gmail.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

librados: don't destroy ObjListCtx when iterator reaches the end

There's no need to destroy it when we've used it once. It'll get destroyed
by the destructor anyway. Now that there's a seek method, we might want seek
and iterate again anyway, which is not possible if the ObjListCtx is gone.

Change get_next() to check the underlying ObjListCtx for the end of iteration
instead of deleting ObjListCtx, and modify the equality operator to count
the end marker (with ctx == NULL) as equal to having a ObjListCtx at the
end of its list.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

librados: remove extra advance from objects_begin(position)

ObjectIterator::seek() already calls get_next() to update the
iterator's state. Calling it again goes one past where it should be.

When the iterator reached the end of the list, it deleted the ListCtx,
causing a segfault on further accesses.

Fixes: #7491
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1286 from ceph/wip-corpus

ceph-object-corpus: prune some old releases

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Revert "ReplicatedPG::recover_backfill: adjust last_backfill to HEAD if snapdir"

This reverts commit 545135f3e1c75318940caa2c82ac32a53fc4f957.

Conflicts:

src/osd/ReplicatedPG.cc

Merge remote-tracking branch 'origin/port/updates' into firefly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

ceph-object-corpus: prune some old releases

We do not need to test these old object encodings as they are covered by
later object collections.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'origin/wip-6936' into firefly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge remote-tracking branch 'origin/wip-7099' into firefly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge remote-tracking branch 'origin/wip-6830' into firefly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge remote-tracking branch 'origin/wip-6951' into firefly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge remote-tracking branch 'origin/wip-7064' into firefly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge remote-tracking branch 'origin/wip-7271' into firefly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

dencoder: check for radosgw build option

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

osd: use ceph scoped shared_ptr

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge remote-tracking branch 'upstream/next'

Merge remote-tracking branch 'upstream/wip-copyget' into next

Fixes: #7471
Reviewed-by: Greg Farnum <greg@inktank.com>

PG::build_might_have_unfound: skip CRUSH_ITEM_NONE

Signed-off-by: Samuel Just <sam.just@inktank.com>