librbd: fast diff of first snapshot loads incorrect object map

The ImageCtx::snaps collection is stored in newest->oldest order,
so the first snapshot id should be retrieved from the back of the
array.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fast diff is incorrectly flagging objects as updated

If the diff_iterate call doesn't start with the first snapshot,
unmodified objects in future snapshots / HEAD are incorrectly
flagged as modified.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4558 from wonzhq/detect-format

librbd: fix the image format detection

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4234 from dmitryya/bug_10163

obj_bencher rados cli: rados bench producing wrong values when different blocksize used in writes and reads

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rados cli: fix documentation for -b option

Fix focumentation for block size option for rados cli

Signed-off-by: Dmitry Yatsushkevich <dyatsushkevich@mirantis.com>

Merge pull request #4527 from SUSE/wip-hack-utf8-into-json-parser

json_spirit: use utf8 intenally when parsing \uHHHH

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4572 from ceph/wip-11540

test_libcephfs: Fix zero length read tests

Reviewed-by: John Spray <john.spray@redhat.com>

test_libcephfs: Fix zero length read tests

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

json_spirit: use utf8 intenally when parsing \uHHHH

When the python CLI is given non-ASCII characters, it converts them to
\uHHHH escapes in JSON.  json_spirit parses these internally into 16 bit
characters, which could only work if json_spirit were built to use
std::wstring, which it isn't; it's using std::string, so the high byte
ends up being zero'd, leaving the low byte which is effectively garbage.

This hack^H^H^H^H change makes json_spirit convert to utf8 internally
instead, which can be stored just fine inside a std::string.

Note that this implementation still assumes \uHHHH escapes are four hex
digits, so it'll only cope with characters in the Basic Multilingual
Plane.  Still, that's rather a lot more characters than it could cope
with before ;)

(For characters outside the BMP, Python seems to generate escapes in the
form \uHHHHHHHH, i.e. 8 hex digits, which the current implementation
doesn't expect to see)

Fixes: #7387
Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #4353 from ceph/wip-11376-packaging-objstore-tool

#11376: packaging: mv ceph-objectstore-tool to main ceph pkg

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4508 from dachary/wip-enoent-offset

osd: For write a non-exist erasure object don't offset > 0.

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #4548 from dachary/wip-docker-ubuntu-12.04

tests: install sudo on ubuntu-12.04 container

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: fix the image format detection

If the detection of the old format fails with reasons other than
-ENOENT, we should return with this error. Otherwise, if we continue the
new format detection and fail with -ENOENT, the caller will get the
missleading failure information.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Merge pull request #4544 from ceph/wip-with-man-pages

Wip with man pages

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4555 from zzxuanyuan/wip-11534-incorrect-osdsum-size

mon: Total size of OSDs is a magnitude less than it is supposed to be.

Reviewed-by: Kefu Chai <kchai@redhat.com>

configure.ac: no use to add "+" before ac_ext=c

Signed-off-by: Kefu Chai <kchai@redhat.com>

configure.ac: add an option: --with-man-pages

* do not require sphinx-build if --without-man-pages
* dist rst files for man pages
* build and dist .8 files if sphinx-build is found

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: Total size of OSDs is a maginitude less than it is supposed to be.

When dumping statistics of OSDs such as running command "ceph osd df",
the sum of OSDs' size is 2^10 times less than their real size.

Signed-off-by: Zhe Zhang <zzxuanyuan@gmail.com>

obj_bencher: aio_bench - rename op_size to object_size

Rename aio_bench argument 'op_size' to 'object_size' to reflect the reality of
how it is used.

Signed-off-by: Dmitry Yatsushkevich <dyatsushkevich@mirantis.com>

rados cli: add preventing using --block-size with bench seq and rand

Add handling of situation when '-b|--block-size' option is using with
'bench seq' or 'bench rand' which is erroneous according to the help message:
     -b op_size
            set the size of write ops for put or benchmarking

Fixes: #10163
Signed-off-by: Dmitry Yatsushkevich <dyatsushkevich@mirantis.com>

obj_bencher: remove excess 'object_size = op_size'

Remove excess branch 'else {object_size = op_size;}' from
'if(operation != OP_WRITE){...}else{object_size = op_size;}' because
'object_size' variable already initialized with 'op_size'

Signed-off-by: Dmitry Yatsushkevich <dyatsushkevich@mirantis.com>

obj_bencher: remove 'trans_size' as obsolete

'trans_size' description from header: "size of the write/read to perform"
But really 'object_size' is used in write/read operations. 'trans_size' is used
only in ObjBencher::status_printer for calc current and average bandwidth.
As result - bad statistics in case 'trans_size' and and 'object_size' are different.

Signed-off-by: Dmitry Yatsushkevich <dyatsushkevich@mirantis.com>

Merge pull request #4554 from liewegas/wip-cleanup

9.0.0 release notes, scripts cleanup

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/release-notes: 9.0.0

Signed-off-by: Sage Weil <sage@redhat.com>

src/script: remove obsolete scripts

These are mostly from gather graph data for very early versions of ceph and
are not useful anymore.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4550 from ktdreyer/wip-submittingpatches-backports

SubmittingPatches: clarify backport procedure

Reviewed-by: Loic Dachary <ldachary@redhat.com>

SubmittingPatches: clarify backport procedure

Developers should not add "Backport: " fields to Git commits, because
this data is immutable after the commits are merged. It makes more sense
to handle this information in Redmine instead.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #4530 from ceph/wip-librbd-clean-object

librbd: object map is not properly updated when deleting clean objects

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tests: install sudo on ubuntu-12.04 container

Contrary to Ubuntu 14.04 it is not installed by default.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: erasure coded pools do not allow offset on creation

If a write operation implicitly creates an object on an erasure coded
pool, there cannot be an offset > 0, even if it is properly aligned.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

osd: refuse to write a new erasure coded object with an offset > 0

Even if the offset is properly aligned.

http://tracker.ceph.com/issues/11507 Fixes: #11507

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Signed-off-by: Loic Dachary <ldachary@redhat.com>

rados: Tell that pool removal failed. Not that it doesn't exist.

If removing a pool fails it could have various reasons. The pool
might be protected from removal (nodelete flag).

Merge pull request #4542 from liewegas/wip-sepia

sepia

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4387 from wonzhq/async-comp

librbd: don't notify_change on r == -ERESTART when notify async complete

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: don't notify_change on error when notify async complete

Don't need to increment the refresh_seq since the async op fails.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Merge pull request #4539 from athanatos/wip-11429

OSD: handle the case where we resurrected an old, deleted pg

Reviewed-by: Sage Weil <sage@redhat.com>

doc: link to sepia report for lab info

Signed-off-by: Sage Weil <sage@redhat.com>

9.0.0

Merge pull request #4537 from majianpeng/buffer-fix

bufferptr: Offset should not beyond raw_length rather than _len.

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

man: do not dist man pages if sphinx is not available

Signed-off-by: Kefu Chai <kchai@redhat.com>

configure.ac: do not check for sphinx-build

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/bufferlist: add test case for bufferlist::splice.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

bufferptr: Offset should not beyond raw_length rather than _len.

Fixes: #11526
Commit dfccd3d19c introduce this bug"
./include/buffer.h: 240: FAILED assert(_len >= o)

 ceph version 0.94-1402-g5ac962d
(5ac962d879d920805bd661857592860caae27f2b)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x80) [0x1640e50]
 2: (ceph::buffer::ptr::set_offset(unsigned int)+0x3a) [0x1647462]
 3: (ceph::buffer::list::splice(unsigned int, unsigned int,
ceph::buffer::list*)+0x238) [0x16458ba]
 4: (JournalStream::read(ceph::buffer::list&, ceph::buffer::list*,
unsigned long*)+0x17c) [0x145588e]
 5: (Journaler::try_read_entry(ceph::buffer::list&)+0x1ef) [0x1453d65]
 6: (MDLog::_replay_thread()+0xc6a) [0x1434200]
 7: (MDLog::ReplayThread::entry()+0x1c) [0x10f0d88]
 8: (Thread::entry_wrapper()+0xa8) [0x16310ee]
 9: (Thread::_entry_func(void*)+0x18) [0x163103c]
 10: /lib64/libpthread.so.0() [0x334c607ee5]
 11: (clone()+0x6d) [0x334bef4d1d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #4514 from majianpeng/rbd

misc bugs in librbd

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4534 from tchaikov/wip-silence-gcc-warning

osd: silence gcc -Wparentheses warning

Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd: silence gcc -Wparentheses warning

otherwise we have following warning:
warning: suggest parentheses around operand of ‘!’ or change ‘|’ to
‘||’ or ‘!’ to ‘~’ [-Wparentheses]

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/CompatSet: mark FeatureSet member vars private

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: For format 2, don't forget set objectcache max objects.

For format 2, before create object_cache, it call init_layout and set
the max object for object cache.
In the later ictx_refresh, it only set for format1.
Now move set logic after creating object cache.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

rbd: updated the 'rbd --help'

Now default format is 2. update 'rbd --help'.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: remove the perfcounter l_librbd_aio_rd/write/discard_*.

Now for non-aio read/write/discard, the internal implementation used the
related aio function. So we don't differentiate the aio perfcounter from non-aio for
read/write/discard operations.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: Remove the redundant judgement.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: Remove the redundant judgement.

For ObjectExtent, if length == object_size, it mean length=0.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Preforker: include acconfig so includers don't have to

Fixes build of systests

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4524 from ceph/wip-remove-fsx-c

test: update CMakefile to sync with c44f8e7

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4518 from yuyuyu101/wip-fix-systest-runnable

systest_runnable: Use Preforker to do fork

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4519 from ceph/wip-flush-log

Flush cluster log before MDS::damaged

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4529 from ceph/wip-librbd-open

librbd: invoking RBD::open twice will leak memory

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4511 from yuyuyu101/wip-async-fix-7

Wip async fix 7

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

librbd: object map is not properly updated when deleting clean objects

With the new fast diff feature flagging which objects haven't changed
between snapshots, unmodified objects are now flagged with a new state
code in the object map.  The OBJECT_EXISTS_CLEAN state needs to be
treated as if it were OBJECT_EXISTS when deleting.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

atomic_t: change cas to compare_and_swap and return bool

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

mds: in damaged() call flush_log before ending

...so that the clog messages we send before damaged()
actually have a chance to make it to the cluster log.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: add MonClient::flush_log

A simple reasonable-effort call for emitting
a MLog before shutting down.  Used in MDS which
also waits for its beacon to be acked, so due to
message ordering implicitly waits for this to land
too -- no need to implement a waiting log flush
for this use case.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #4516 from yuyuyu101/wip-fix-libcephfs-crash

Wip fix libcephfs crash

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

librbd: invoking RBD::open twice will leak memory

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4492 from SUSE/wip-suse-firewall

packaging: add SuSEfirewall2 service files

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4468 from cernceph/swifty

Swift: Set Content-Length when requesting/checking Keystone tokens

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4008 from SUSE/wip-tcmalloc-conditionals

Conditional-compile against minimal tcmalloc.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3429 from liewegas/wip-precalc-pgtemp

RFC: mon: prime pg_temp when osdmap changes

Reviewed-by: Guang Yang <yguang@yahoo-inc.com>
Tested-by: Zhi (David) Zhang <zhangz@yahoo-inc.com>

Merge branch 'wip-journal-header' of git://github.com/XinzeChi/ceph

Conflicts:
src/gmock

Merge pull request #3474 from majianpeng/fix5

buffer: optimize get_contiguous

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3896 from ceph/wip-auth

auth: optimize crypto key handling, msg sig checks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3975 from wonzhq/skip-cache

osd: flush/evict op fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4362 from XinzeChi/wip-tier-lat

osd: misc latency perf counter for tier pool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4393 from majianpeng/copyget-flags

For CEPH_OSD_OP_COPY_GET, add a new filed: flags.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3564 from guangyy/wip-10198

osd: break the handling of PG removal event into multiple iterations

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3004 from guangyy/wip-10183

Add a new tick timer which does not need to hold osd_lock, put sched_scrub to that tick timer

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3659 from SUSE/wip-fix-calamari-health

Always provide summary for non-healthy cluster.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3702 from mslovy/wip-fineGrain-submitManagerLock

FileStore:: queue_transactions, fine-grain submitManager lock

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3919 from guangyy/wip-9558

osd : populate the needs_recovery_map source fast when only one peer has missing

Reviewed-by: Samuel Just <sjust@redhat.com>

OSD: handle the case where we resurrected an old, deleted pg

Prior to giant, we would skip pgs in load_pgs which were not present in
the current osdmap.  Those pgs would eventually refer to very old
osdmaps, which we no longer have causing the assertion failure in 11429
once the osd is finally upgraded to a version which does not skip the
pgs.  Instead, if we do not have the map for the pg epoch, complain to
the osd log and skip the pg.

Fixes: 11429
Signed-off-by: Samuel Just <sjust@redhat.com>

test: update CMakefile to sync with c44f8e7

* remove unused src/test/fsx.c, see
  qa/workunits/suites/fsx.sh

Signed-off-by: Kefu Chai <kchai@redhat.com>

systest_runnable: adjust argument to suite Preforker

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

ceph_mon: output preforker error message

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Preforker: Add child process exit status check

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

systest_runnable: Use Preforker to do fork

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

test_libcephfs: Add tests to cover zero length write

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

packaging: mv ceph-objectstore-tool to main ceph pkg

This change ensures that the ceph-objectstore-tool utility is present on
all OSDs.  This makes it easier for users to run this tool to do manual
debugging/recovery in some scenarios.

http://tracker.ceph.com/issues/11376 Refs: #11376

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #4493 from ceph/wip-use-get-linux-version

use get_linux_version() instead ad-hoc uname().release parsing

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Client: Fast return if len is 0

If len == 0,
osdc/Striper.cc: 50: FAILED assert(len > 0)
 2: (Striper::file_to_extents(CephContext*, char const*, ceph_file_layout const*, unsigned long, unsigned long, unsigned long, std::map<object_t, std::vector<ObjectExtent, std::allocator<ObjectExtent> >, std::less<object_t>, std::allocator<std::pair<object_t const, std::vector<ObjectExtent, std::allocator<ObjectExtent> > > > >&, unsigned long)+0x1c6) [0x7fd1a0f4f522]
 3: (Striper::file_to_extents(CephContext*, char const*, ceph_file_layout const*, unsigned long, unsigned long, unsigned long, std::vector<ObjectExtent, std::allocator<ObjectExtent> >&, unsigned long)+0x69) [0x7fd1a0f4f315]
 4: (Striper::file_to_extents(CephContext*, inodeno_t, ceph_file_layout const*, unsigned long, unsigned long, unsigned long, std::vector<ObjectExtent, std::allocator<ObjectExtent> >&)+0x8a) [0x7fd1a0ef9c3c]
 5: (ObjectCacher::file_read(ObjectCacher::ObjectSet*, ceph_file_layout*, snapid_t, long, unsigned long, ceph::buffer::list*, int, Context*)+0x7e) [0x7fd1a0efb186]
 6: (Client::_read_async(Fh*, unsigned long, unsigned long, ceph::buffer::list*)+0x4ca) [0x7fd1a0ec90d4]
 7: (Client::_read(Fh*, long, unsigned long, ceph::buffer::list*)+0x3dc) [0x7fd1a0ec8744]
 8: (Client::read(int, char*, long, long)+0x208) [0x7fd1a0ec8100]
 9: (ceph_read()+0x5e) [0x7fd1a0e80603]

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4461 from ceph/wip-11372

librbd: add valgrind memory checks for unit tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4241 from rzarzynski/wip-11285

rgw: rectify broken statistics on Swift account listing.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4432 from big-henry/wip-rgw-listparts

rgw: fix ListParts response

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

pybind: fix valgrind warning on rbd_get_parent_info call

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osdc: invalid read of freed memory

The bytes not in cache stat was potentially reading the bh length
from a deleted bufferhead.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

krbd: fix incorrect types in the krbd API

The C API functions were referencing the C++ CephContext
instead of the C rados_config_t.  Additionally, the ceph
namespace was missing on the Formatter class.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

fsx: cleanup crypto library at exit

Also made small tweaks so that it can be compiled under
a C++ compiler.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

autotools: detect presence of valgrind

Conditionally enable running librbd unit tests under valgrind.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: add support for running API tests under valgrind

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: run librbd valgrind test during 'make check'

Fixes: #11372
Signed-off-by: Jason Dillaman <dillaman@redhat.com>