rbd: fix cli test

Signed-off-by: Sage Weil <sage@inktank.com>

rgw: quiet down warning message

Fixes: #6123
We don't want to know about failing to read region map info
if it's not found, only if failed on some other error. In
any case it's just a warning.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

rbd.cc: add readonly option for "rbd map"

The device could only be set to rw(default) when mapping
now. This patch only handle the user space, because the
kernel part has been completed.

Signed-off-by: Guangliang Zhao <guangliang@unitedstack.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #660 from ceph/wip-fs-crc

sloppy / opportunistic CRC tracking in the filestore

Reviewed-by: Samuel Just <sam.just@inktank.com>

os/FileStore: add sloppy crc tracking

Opportunistically track CRCs for data we write and verify it for data
we read.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #661 from ceph/wip-copy-cleanups

Wip copy cleanups

Reviewed-by: Sage Weil <sage@inktank.com>

ReplicatedPG: remove unused CopyOp::waiting member, rename function for accuracy

We never placed any ops here, and so we weren't ever requeuing any ops.

Signed-off-by: Greg Farnum <greg@inktank.com>

ReplicatedPG: remove useless pcop param from start_copy

Signed-off-by: Greg Farnum <greg@inktank.com>

ReplicatedPG: use our already-found iterator instead of going back into map

We have an iterator pointing at the element we want; no need to search
the map again in order to grab the element or remove it.

Signed-off-by: Greg Farnum <greg@inktank.com>

common/SloppyCRCMap: add type to sloppily track crcs

Signed-off-by: Sage Weil <sage@inktank.com>
sloppy

Merge remote-tracking branch 'gh/next'

Conflicts:
src/common/crc32c_intel_fast.c

Merge pull request #658 from dmick/next

ceph.in: fix missing exception variable in failure to open -o file

Reviewed-by: Sage Weil <sage@inktank.com>

ceph.in: fix missing exception variable in failure to open -o file

Fixes: #6424
Signed-off-by: Dan Mick <dan.mick@inktank.com>

Merge pull request #657 from ceph/wip-6397

Wip 6397

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

rgw: don't append NULL char before json parsing

Parser handles that by itself.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

ceph_json: use different string constructor for parser buffer

Fixes: #6397
Previous implementation did not user the constructor with the length
param.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #643 from dachary/master

autoconf: fix build out of tree

Reviewed-by: Sage Weil <sage@inktank.com>

doc: changed journal aio default to true.

fixes: #6389

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

qa: workunits: mon: test snaps ops using rbd.

Regression test for #6047

Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
(cherry picked from commit b2b0f202ea8a0028e7e27664b3ea6b9b8cdafe6e)

Merge pull request #654 from ceph/wip-testrados

fix some issues with ceph_test_rados

Reviewed-by: Samuel Just <sam.just@inktank.com>

ceph_test_rados: remove useless snapc setup

We update this on snap create and delete; no need to do it any other
time.

Signed-off-by: Sage Weil <sage@inktank.com>

ceph_test_rados: update for copy_from in begin, not finish

Signed-off-by: Sage Weil <sage@inktank.com>

common/crc32c_intel_fast: avoid reading partial trailing word

The optimized intel code reads in word-sized chunks, knowing that the
allocator will only hand out memory in word-sized increments.   This makes
valgrind unhappy.  Whitelisting doesn't work because for some reason there
is no caller context (probably because of some interaction with yasm?).

Instead, just use the baseline code for the last few bytes.  This should
not be significant.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 39c89dcfed0587f822e2226f2de8e4092449af29)

Conflicts:
src/test/common/test_crc32c.cc

Merge pull request #650 from dachary/wip-6425-2

pybind: update osd blacklist tests for nonce

pybind: update osd blacklist tests for nonce

Update the unit tests to reflect a) that the nonce is no longer mandatory,
b) that it must be an integer.

http://tracker.ceph.com/issues/6426 refs #6426

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

test/common/get_command_descriptions: null terminate c string

This fixes a bunch of trailing goop I see on my machine.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/next'

Conflicts:
src/pybind/ceph_argparse.py

Merge pull request #648 from dmick/wip-6425

ceph_argparse.py, cephtool/test.sh: fix blacklist with no nonce

Reviewed-by: Sage Weil <sage@inktank.com>

ceph_test_rados: improve debug output

Signed-off-by: Sage Weil <sage@inktank.com>

ceph_test_rados: fix update_object_full

Update at current snap, not the last snap we did an update.

Broken this this was introduced in 996af2d8fd8e60bcdce8e9408249b974521de24e

Signed-off-by: Sage Weil <sage@inktank.com>

ceph_argparse.py, cephtool/test.sh: fix blacklist with no nonce

It's legal to give a CephEntityAddr to osd blacklist with no nonce,
so allow it in the valid() method; also add validation of any nonce
given that it's a long >= 0.

Also fix comment on CephEntityAddr type description in MonCommands.h,
and add tests for invalid nonces (while fixing the existing tests to remove
the () around expect_false args).

Fixes: #6425
Signed-off-by: Dan Mick <dan.mick@inktank.com>

Merge pull request #646 from liewegas/wip-6385

qa/run_xfstests.sh: use old xfstests until we adapt to new org

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #647 from ceph/wip-ceph-cli-env

improve ceph clie CEPH_ARGS handling

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Revert "ceph: parse CEPH_ARGS environment variable"

This reverts commit 67a95b9880c9bc6e858150352318d68d64ed74ad.

We now put CEPH_ARGS in the actual args we parse in python, which are passed
to rados piecemeal later.  This lets you put things like --id ... in there
that need to be parsed before librados is initialized.

Add CEPH_ARGS at the end of sys.argv

This allows, for instance, to pass a different client name to ceph by
exporting CEPH_ARGS="--id client_id".

Signed-off-by: Benoît Knecht <benoit.knecht@fsfe.org>
Signed-off-by: Sage Weil <sage@inktank.com>

qa/run_xfstests.sh: use old xfstests until we adapt to new org

Tests were rearranged upstream; use an old version for the time being
until we can refactor run_xfstests.sh to cope.  See #6385

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #645 from liewegas/wip-6346

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #631 from ceph/wip-5857-8

Wip 5857 8

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #546 from ceph/wip-5862

Wip 5862

Reviewed-by: Samuel Just <sam.just@inktank.com>

os: Make write_version_stamp() private

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd: Remove code for reading ancient unsupported logs

Signed-off-by: David Zafman <david.zafman@inktank.com>

os/ObjectStore: Interim collection_list* functions in ObjectStore

Handle ghobject_t to hobject_t conv of collection_list* funcs
Temporary code so that this branch doesn't break master

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd: Cleanup init()/read_superblock()

Fix error handling in init()
Cleanup read_superblock() by moving unrelated code into init()
Move init() feature upgrade right after compatibility checking
Remove redundant whoami check

Signed-off-by: David Zafman <david.zafman@inktank.com>

common, os, osd, test, tools: FileStore must work with ghobjects rather than hobjects

Add ghobject_t to hboject.h header
Add constants NO_SHARD/NO_GEN and change gen_t/shard_t
Convert other headers from hobject_t to ghobject_t
Mostly straight hobject_t to ghobject_t for src/os cc files
Fix tools and tests and enable ceph-dencoder
Add filename generation and parsing including unittest addition
Get ceph-filestore-dump to build
Add gen/shard to DBObjectMap::ghobject_key() and update test case
Add CEPH_FS_FEATURE_INCOMPAT_SHARDS new FileStore feature
Add CEPH_OSD_FEATURE_INCOMPAT_SHARDS new osd feature

Fixes: #5862
Signed-off-by: David Zafman <david.zafman@inktank.com>

include, test: Add CompatSet::merge() operation

New func merge() adds missing features from specified CompatSet
Add merge testing to unittest

Signed-off-by: David Zafman <david.zafman@inktank.com>

os, osd, tools: Add backportable compatibility checking for sharded objects

OSD
  New CEPH_OSD_FEATURE_INCOMPAT_SHARDS
FileStore
  NEW CEPH_FS_FEATURE_INCOMPAT_SHARDS
  Add FSSuperblock with feature CompatSet in it
  Store sharded_objects state using CompatSet
  Add set_allow_sharded_objects() and get_allow_sharded_objects() to FileStore/ObjectStore
  Add read_superblock()/write_superblock() internal filestore functions
ceph_filestore_dump
  Add OSDsuperblock to export format
  Use CompatSet from OSD code itself in filestore-dump tool
  Always check compatibility of OSD features with on-disk features
  On import verify compatibility of on-disk features with export data
  Bump super_ver due to export format change

Backport: dumpling, cuttlefish

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd: read_superblock() not outputing unsupported features when incompatible

Use working version of CompatSet::unsupported() to improve log output

Backport: dumpling, cuttlefish

Signed-off-by: David Zafman <david.zafman@inktank.com>

test: Adding unittest for CompatSet

Signed-off-by: David Zafman <david.zafman@inktank.com>

include: Bug fixes for CompatSet

FeatureSet insert/remove
  Use 64-bit arithmetic to allow features past 31
  Allow feature 63 by fixing assert in insert
CompatSet::unsupported() bugs
  Ignore feature 0 which became illegal
  Use 64-bit arithmetic when computing mask
  Use id in insert() and to get correct feature name
  Use the right map to get name for diff.ro_compat

Caused by 80979bbe92409e6f098566e18be6ed59ad9d111a
Caused by 470796b5456592ee5179bbd44b72910a2d7f6aca

Backport: dumpling, cuttlefish

Signed-off-by: David Zafman <david.zafman@inktank.com>

os: Fix version_stamp_is_valid() and write_version_stamp() error return

Caused by 341fb208aa32117f58ed4ce623f4b152f64ac72e

Signed-off-by: David Zafman <david.zafman@inktank.com>

ReplicatedPG: take obc read lock prior to recovering an object to replicas

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend::handle_pull_response: use list<ObjectContextRef>

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: remove noop C_OSD_CompletedPull

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: assert build_push_op succeeds on primary

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG.h: remove temp_coll stub

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD,ReplicatedPG: queue pushes in a op_tp workqueue

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: don't proceed to backfill until recovering is empty

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: correctly handle backfill

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: fix up recalc_subsets

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: wire in start_pushes

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: handle stats properly

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: track object contexts in Push/PullInfo

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: remove priority from Pull/PushInfo

Instead, we'll just nab the priority from the messages.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: move prep_push and friends to ReplicatedBackend

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: remove pushing/pulling

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: handle down pull sources

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: split recover_missing out of prepare_pull

Also, move prepare_pull to PGBackend.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: shuffle over submit_push_*/build_push_op

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: move over send_pushes/send_pulls/prep_push_op_blank

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: move over _failed_push

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: move handle_*

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: shuffle do_pull/do_push_reply

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: move _do_push,_do_pull_response,do_push

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: convert sub_op_pull

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: make log_subop_stats static

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: shuffle sub_op_push_reply

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: shuffle sub_op_push over to ReplicatedBackend

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: allow dout for ReplicatedBackend methods to work in ReplicatedPG.cc

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: extract PGBackend::Listener recovery callbacks

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG,ReplicatedPG: handle do_request in ReplicatedPG,PGBackend

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: pass on dump_recovery_info to PGBackend

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD,ReplicatedPG: let PGBackend handle the temp collection

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: add PGBackend interfaces and stubs

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: Allow get_object_context caller to provide attributes

This will be used by PGBackend implementers to get the SnapSet
and ObjectInfo for newly recovered objects on the primary.

get_object_context may be called on a missing object for a lost_revert

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: send_push is unused

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore: ping tphandle between each operation within a transaction

Signed-off-by: Samuel Just <sam.just@inktank.com>

WorkQueue: add a workqueue which simply runs queued GenContexts

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #644 from ceph/wip-6345

reverse the order of the mons in the init script when stopping

Reviewed-by: Sage Weil <sage@inktank.com>

osd/ReplicatedPG: fix bl resize on write vs truncate race

If we resize the write due to the funky truncate behavior, we need to
resize the bufferlist to match.

Fixes: #6346
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #606 from ceph/wip-6332

Reviewed-by: Sage Weil <sage@inktank.com>

mon: OSDMonitor: do not write full_latest during trim

On commit 81983bab we patched OSDMonitor::update_from_paxos() such that we
write the latest full map version to 'full_latest' each time the latest
full map was built from the incremental versions.

This change however clashed with OSDMonitor::encode_trim_extra(), which
also wrote to 'full_latest' on each trim, writing instead the version of
the *oldest* full map.  This duality of behaviors could lead the store
to an inconsistent state across the monitors (although there's no sign of
it actually imposing any issues besides rebuilding already existing full
maps on some monitors).

We now stop OSDMonitor::encode_trim_extra() from writing to 'full_latest'.
This function will still write out the oldest full map it has in the store,
but it will no longer write to full_latest, instead leaving it up to
OSDMonitor::update_from_paxos() to figure it out -- and it already does.

Fixes: #6378
Backport: dumpling

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

mon: don't list "--yes-i-really-mean-it" options as required

By default options are required, and that leads to the Ceph CLI making
users add this to their commands before even submitting it to the monitor. We
want them to see and think about the messages generated by the monitor!

Signed-off-by: Greg Farnum <greg@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge remote-tracking branch 'gh/next'

do not quote the variable to loop over

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

msg/msg_types: use proper NI_MAXSERV when formatting an IP address

May as well be pedantic about it, even though we are leaving the port
in numeric form.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

reverse the order of the mons in the init script when stopping

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

qa: update snaps workunits for allow_new_snaps flag

Test that it works in snaptest-0.sh, and set the flag in
all the snap workunits so they continue to function.

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

MDS: lock out snapshots until after a flag has been set in the MDSMap

This way users can't put snapshots on their clusters unless they explicitly
ask for them and have seen a warning message. We take a bit of the MDSMap
flags in order to do so. The only thing a little weird here is that anybody
who upgrades to this patch who already has snapshots will hit the EPERM
and have to go through the warning, but it doesn't impact existing snapshots
at all so they should be good.

To go along with this, we add "ever_allowed_snaps" and "explicitly_allowed_snaps"
members to the MDSMap, which are default to false and are set to true
when allow_new_snaps is set. Old maps decoded with new code default to true
and false, respectively, so we can tell.

Fixes: #6332
Signed-off-by: Greg Farnum <greg@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

mds: properly return log replay error

The only context that should see this is
C_MDS_BootStart, and it is prepared for failures,
so there's no reason to always return 0.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

common/crc32c_intel_fast: fix compile-time #ifdef

This wasn't getting built in!

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit 3233336cc3b6c2c1e89fe6c6d21d42e0f2cce142)