ceph_test_rados: remove useless snapc setup

We update this on snap create and delete; no need to do it any other
time.

Signed-off-by: Sage Weil <sage@inktank.com>

ceph_test_rados: update for copy_from in begin, not finish

Signed-off-by: Sage Weil <sage@inktank.com>

ceph_test_rados: improve debug output

Signed-off-by: Sage Weil <sage@inktank.com>

ceph_test_rados: fix update_object_full

Update at current snap, not the last snap we did an update.

Broken this this was introduced in 996af2d8fd8e60bcdce8e9408249b974521de24e

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #631 from ceph/wip-5857-8

Wip 5857 8

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #546 from ceph/wip-5862

Wip 5862

Reviewed-by: Samuel Just <sam.just@inktank.com>

os: Make write_version_stamp() private

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd: Remove code for reading ancient unsupported logs

Signed-off-by: David Zafman <david.zafman@inktank.com>

os/ObjectStore: Interim collection_list* functions in ObjectStore

Handle ghobject_t to hobject_t conv of collection_list* funcs
Temporary code so that this branch doesn't break master

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd: Cleanup init()/read_superblock()

Fix error handling in init()
Cleanup read_superblock() by moving unrelated code into init()
Move init() feature upgrade right after compatibility checking
Remove redundant whoami check

Signed-off-by: David Zafman <david.zafman@inktank.com>

common, os, osd, test, tools: FileStore must work with ghobjects rather than hobjects

Add ghobject_t to hboject.h header
Add constants NO_SHARD/NO_GEN and change gen_t/shard_t
Convert other headers from hobject_t to ghobject_t
Mostly straight hobject_t to ghobject_t for src/os cc files
Fix tools and tests and enable ceph-dencoder
Add filename generation and parsing including unittest addition
Get ceph-filestore-dump to build
Add gen/shard to DBObjectMap::ghobject_key() and update test case
Add CEPH_FS_FEATURE_INCOMPAT_SHARDS new FileStore feature
Add CEPH_OSD_FEATURE_INCOMPAT_SHARDS new osd feature

Fixes: #5862
Signed-off-by: David Zafman <david.zafman@inktank.com>

include, test: Add CompatSet::merge() operation

New func merge() adds missing features from specified CompatSet
Add merge testing to unittest

Signed-off-by: David Zafman <david.zafman@inktank.com>

os, osd, tools: Add backportable compatibility checking for sharded objects

OSD
  New CEPH_OSD_FEATURE_INCOMPAT_SHARDS
FileStore
  NEW CEPH_FS_FEATURE_INCOMPAT_SHARDS
  Add FSSuperblock with feature CompatSet in it
  Store sharded_objects state using CompatSet
  Add set_allow_sharded_objects() and get_allow_sharded_objects() to FileStore/ObjectStore
  Add read_superblock()/write_superblock() internal filestore functions
ceph_filestore_dump
  Add OSDsuperblock to export format
  Use CompatSet from OSD code itself in filestore-dump tool
  Always check compatibility of OSD features with on-disk features
  On import verify compatibility of on-disk features with export data
  Bump super_ver due to export format change

Backport: dumpling, cuttlefish

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd: read_superblock() not outputing unsupported features when incompatible

Use working version of CompatSet::unsupported() to improve log output

Backport: dumpling, cuttlefish

Signed-off-by: David Zafman <david.zafman@inktank.com>

test: Adding unittest for CompatSet

Signed-off-by: David Zafman <david.zafman@inktank.com>

include: Bug fixes for CompatSet

FeatureSet insert/remove
  Use 64-bit arithmetic to allow features past 31
  Allow feature 63 by fixing assert in insert
CompatSet::unsupported() bugs
  Ignore feature 0 which became illegal
  Use 64-bit arithmetic when computing mask
  Use id in insert() and to get correct feature name
  Use the right map to get name for diff.ro_compat

Caused by 80979bbe92409e6f098566e18be6ed59ad9d111a
Caused by 470796b5456592ee5179bbd44b72910a2d7f6aca

Backport: dumpling, cuttlefish

Signed-off-by: David Zafman <david.zafman@inktank.com>

os: Fix version_stamp_is_valid() and write_version_stamp() error return

Caused by 341fb208aa32117f58ed4ce623f4b152f64ac72e

Signed-off-by: David Zafman <david.zafman@inktank.com>

ReplicatedPG: take obc read lock prior to recovering an object to replicas

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend::handle_pull_response: use list<ObjectContextRef>

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: remove noop C_OSD_CompletedPull

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: assert build_push_op succeeds on primary

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG.h: remove temp_coll stub

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD,ReplicatedPG: queue pushes in a op_tp workqueue

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: don't proceed to backfill until recovering is empty

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: correctly handle backfill

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: fix up recalc_subsets

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: wire in start_pushes

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: handle stats properly

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: track object contexts in Push/PullInfo

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: remove priority from Pull/PushInfo

Instead, we'll just nab the priority from the messages.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: move prep_push and friends to ReplicatedBackend

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: remove pushing/pulling

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: handle down pull sources

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: split recover_missing out of prepare_pull

Also, move prepare_pull to PGBackend.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: shuffle over submit_push_*/build_push_op

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: move over send_pushes/send_pulls/prep_push_op_blank

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: move over _failed_push

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: move handle_*

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: shuffle do_pull/do_push_reply

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: move _do_push,_do_pull_response,do_push

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: convert sub_op_pull

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: make log_subop_stats static

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: shuffle sub_op_push_reply

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: shuffle sub_op_push over to ReplicatedBackend

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: allow dout for ReplicatedBackend methods to work in ReplicatedPG.cc

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: extract PGBackend::Listener recovery callbacks

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG,ReplicatedPG: handle do_request in ReplicatedPG,PGBackend

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: pass on dump_recovery_info to PGBackend

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD,ReplicatedPG: let PGBackend handle the temp collection

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: add PGBackend interfaces and stubs

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: Allow get_object_context caller to provide attributes

This will be used by PGBackend implementers to get the SnapSet
and ObjectInfo for newly recovered objects on the primary.

get_object_context may be called on a missing object for a lost_revert

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: send_push is unused

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore: ping tphandle between each operation within a transaction

Signed-off-by: Samuel Just <sam.just@inktank.com>

WorkQueue: add a workqueue which simply runs queued GenContexts

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #644 from ceph/wip-6345

reverse the order of the mons in the init script when stopping

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #606 from ceph/wip-6332

Reviewed-by: Sage Weil <sage@inktank.com>

mon: don't list "--yes-i-really-mean-it" options as required

By default options are required, and that leads to the Ceph CLI making
users add this to their commands before even submitting it to the monitor. We
want them to see and think about the messages generated by the monitor!

Signed-off-by: Greg Farnum <greg@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge remote-tracking branch 'gh/next'

do not quote the variable to loop over

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

msg/msg_types: use proper NI_MAXSERV when formatting an IP address

May as well be pedantic about it, even though we are leaving the port
in numeric form.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

reverse the order of the mons in the init script when stopping

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

qa: update snaps workunits for allow_new_snaps flag

Test that it works in snaptest-0.sh, and set the flag in
all the snap workunits so they continue to function.

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

MDS: lock out snapshots until after a flag has been set in the MDSMap

This way users can't put snapshots on their clusters unless they explicitly
ask for them and have seen a warning message. We take a bit of the MDSMap
flags in order to do so. The only thing a little weird here is that anybody
who upgrades to this patch who already has snapshots will hit the EPERM
and have to go through the warning, but it doesn't impact existing snapshots
at all so they should be good.

To go along with this, we add "ever_allowed_snaps" and "explicitly_allowed_snaps"
members to the MDSMap, which are default to false and are set to true
when allow_new_snaps is set. Old maps decoded with new code default to true
and false, respectively, so we can tell.

Fixes: #6332
Signed-off-by: Greg Farnum <greg@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

mds: properly return log replay error

The only context that should see this is
C_MDS_BootStart, and it is prepared for failures,
so there's no reason to always return 0.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

common/crc32c_intel_fast: fix compile-time #ifdef

This wasn't getting built in!

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit 3233336cc3b6c2c1e89fe6c6d21d42e0f2cce142)

Merge pull request #632 from ceph/wip-rworder

fix up read vs write ordering flags; make ceph_test_rados use it

Reviewed-by: Greg Farnum <greg@inktank.com>

Context: add GenContext templated on the callback argument

Signed-off-by: Samuel Just <sam.just@inktank.com>

mds/Server: fix LOOKUPSNAP

The current LOOKUPSNAP code path was dereferencing mdr->dn[0] and getting
SEGV.  Instead, set the do_lookup arg to false for getattr so that we do
not try to dereference this; for snaps it will always be NULL as we are
constructing a fake namespace for the .snap directory.  (This happens in
the path_traverse, the result of which is to set snapid--without any real
dentry).

With this fix, snaptest-0.sh passes.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #640 from ceph/wip-mon-pgmap

fix mon crash when osdmap and pgmap aren't yet synced

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon/PGMonitor: fix segfault when osdmap and pgmap stats are out of sync

The leader mon is responsible for propagating changes in the osdmap
into the pgmap (e.g., when a pool or osd goes away).  If that hasn't
happened yet, the pg_map map will be out of sync with the current set
of pools.  We shouldn't crash.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #637 from dachary/master

mailmap: add Matthew Roy, Matthew Wodrich

Merge pull request #639 from ceph/wip-gtest

Makefile: don't use srcdir in TESTS

Reviewed-by: Roald J. van Loon <roaldvanloon@gmail.com>

Makefile: don't use srcdir in TESTS

src/Makefile-env.am:31: error: using '$(srcdir)' in TESTS is currently broken: '$(srcdir)/unittest_bufferlist.sh'
src/Makefile.am:1:   'src/Makefile-env.am' included from here
src/Makefile-env.am:31: error: using '$(srcdir)' in TESTS is currently broken: '$(srcdir)/test/encoding/check-generated.sh'
src/Makefile.am:1:   'src/Makefile-env.am' included from here

on my (raring) laptop.

Signed-off-by: Sage Weil <sage@inktank.com>

mailmap: add Matthew Roy, Matthew Wodrich

Signed-off-by: Loic Dachary <loic@dachary.org>

qa/workunits/mon/crush_ops.sh: fix test

Fix root.

Fixes: #6392
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #615 from dachary/wip-jerasure-copy

ErasureCode: get rid of extra copy when encoding

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #634 from dachary/master

git: consolidate mails in commit logs

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #636 from dachary/wip-get_command_description

autoconf: remove get_command_descriptions  dependency to gtest

Reviewed-by: Sage Weil <sage@inktank.com>

ErasureCode: get rid of extra copy when encoding

The substr_of bufferlist method is used to reduce the copies to the
strict minimum. It is both able to split bufferptr when they are larger
than a chunk and to concatenate them when they are smaller.

Signed-off-by: Loic Dachary <loic@dachary.org>

autoconf: remove get_command_descriptions  dependency to gtest

It is not required and a leftover of the time when it was a unit test
which was a mistake to begin with. It has been reported to create
dependencies problems in some cases, when gest is not available at the
time get_command_descriptions is compiled.

Signed-off-by: Loic Dachary <loic@dachary.org>

git: consolidate mails in commit logs

Map duplicate mails / names in commit logs to a single canonical names.
Useful to report to management with git shortlog -nes

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge remote-tracking branch 'gh/next'

Conflicts:
src/osd/ReplicatedPG.cc

Merge pull request #633 from ceph/wip-repop-leaks

fix a copule of leaks of RepGather structs

Reviewed-by: David Zafman <david.zafman@inktank.com>

doc: Removed service ops language and added reference to operations.

fixes: #6374

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

mon/OSDMonitor: fix 'ceph osd crush reweight ...'

The adjust method returns a count of adjusted items.

Add a test.

Fixes: #6382
Backport: dumpling
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>

Merge pull request #477 from jpfrancois/patch-1

Fix for Swift key command

osd/ReplicatedPG: fix leak of RepGather on watch timeout

new_repop() gives us a ref; we need to release it, just as we do in
execute_ctx().

Backport: dumpling
Signed-off-by: Sage Weil <sage@inktank.com>

osd/ReplicatedPG: fix leak of RepGather on large-object COPY_FROM

new_repop() gives us a ref; we need to release it, just as we do in
execute_ctx().

Signed-off-by: Sage Weil <sage@inktank.com>

ceph_test_rados: order racing read wrt to the COPY_FROM

The prevents:

 - send COPY_FROM, blocks on recovering object
 - send stat, procedes to return old object version
 - COPY_FROM succeeds with new version
   -> fail

Signed-off-by: Sage Weil <sage@inktank.com>

librados: add OPERATION_ORDER_READS_WRITES flag

This maps to the internal RWORDERED option, which indicates that a read
should be ordered the same way a write is.

Signed-off-by: Sage Weil <sage@inktank.com>

osd/ReplicatedPG: respect RWORDERED rados flag

If this flag is set, we need to order reads as writes.  In particular, this
means that reads will wait for degraded object recovery even if there is a
local copy.  And subsequently will be ordered after a preceding write that
is waiting for the same thing.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: change warn_interval_multiplier to uint32_t

to prevent overflow in OpTracker::check_ops_in_flight when
multiplying warn_interval_multiplier *= 2

Backport: cuttlefish, dumpling

http://tracker.ceph.com/issues/6370 fixes #6370

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 1bce1f009bffd3e28025a08775fec189907a81db)

Merge pull request #630 from dachary/master

osd: change warn_interval_multiplier to uint32_t

Reviewed-by: Sage Weil <sage@inktank.com>

arch/intel: fix old comment

Signed-off-by: Sage Weil <sage@inktank.com>

arch/intel: use intel probe instructions for x86_64 only

Not LP64, which includes ppc64 and clearly does not build.

Fixes: #6283
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

osd: change warn_interval_multiplier to uint32_t

to prevent overflow in OpTracker::check_ops_in_flight when
multiplying warn_interval_multiplier *= 2

Backport: cuttlefish, dumpling

http://tracker.ceph.com/issues/6370 fixes #6370

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #628 from dachary/wip-erasure-fix-thread-test

ErasureCode: fix plugin loading threaded test

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #627 from dachary/master

rgw: fix rgw test to reflect usage change

Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

ErasureCode: fix plugin loading threaded test

The TEST_F(ErasureCodePluginRegistryTest, factory_mutex) was bugous and
only succeeded by chance. The sleep was on the factory constructor which
was never called. An erasure code plugin that hangs forever on load is
created instead.

The sleep_forever.detach is replaced by pthread_cancel to interrupt the
thread that hangs forever. If not, gtest will try to join the thread and
never exit.

Signed-off-by: Loic Dachary <loic@dachary.org>

rgw: fix rgw test to reflect usage change

otherwise src/test/cli/radosgw-admin/help.t fails when running make
check when run after a configure --with-radosgw

Signed-off-by: Loic Dachary <loic@dachary.org>