python-rados: Add object lock support

This change adds to the python binding the support of:
 - rados_lock_exclusive
 - rados_lock_shared
 - rados_unlock

http://tracker.ceph.com/issues/6114 Refs: #6114

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

common/ceph_context: don't import std namespace

This was broken by 7fed5dee4f96a83d1d6914f6fc0895bba2d15b99

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3020 from dachary/wip-10192-objectstore-tool-lookup

objectstore tool lookup by name & --op list filter

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: Accept json object with --pgid instead of array

It isn't anticipated that anyone would use this but keeps backward compatible

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Improve object spec parsing error messages

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Fix errors messages in newer code

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Remove extraneous endl on error throw messages

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Add --format and --pretty-format support

--pretty-format defaults true
Add --format so xml output can be requested
--op list defaults to single line of json per object
To override this more human readable output use --pretty-format=false
Add testing of --op list special handling

Signed-off-by: David Zafman <dzafman@redhat.com>

.gitignore: add ceph_perf_objectstore

Signed-off-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2987 from yuyuyu101/wip-9888

AsyncMessenger: Async event threads can shared by all AsyncMessenger

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3043 from yuyuyu101/objectstore-perf

Objectstore::Transaction Perf Program

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3094 from yuandong1222/bufferlist-get_contiguous

common: bufferlist::get_contiguous return 0 when param len == 0

Reviewed-by: Sage Weil <sage@redhat.com>

common: bufferlist::get_contiguous return 0 when param len == 0

Change-Id: I3eaee45b12c634687cd1b294b3c264aed8cab03a
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

ceph_objectstore_tool: Strip _* (always _head) from pgid in list entry output

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: BUG: --op list wasn't including snapshots

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: For terminal output of list one object per line

Instead of a parsable array make it easier to cut and paste listed objects

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: In error case umount objectstore

Signed-off-by: David Zafman <dzafman@redhat.com>

objectstore_tool: test --op list variants

Signed-off-by: Loic Dachary <ldachary@redhat.com>

objectstore_tool: parse new object description format

The object format changed from

    {json object}

to

    [pgid,{json object}]

The parser is updated accordingly. If the --pgid is present, check that
it equals the pgid from the object description.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

objectstore_tool: filter --op list and explore all PGs

The positional object name is used to filter the output of --op list and
only show the objects with a matching name. If both the object name and
the pgid are omitted, all objects from all PGs are displayed.

The output format is changed from

    {"oid":"GROUP","key":"","snapid":-2,
     "hash":2659194943,"max":0,"pool":0,"namespace":""}

to

    [["0.7_head",{"oid":"GROUP","key":"","snapid":-2,
                  "hash":2659194943,"max":0,"pool":0,
                  "namespace":""}]]

where the first member is the pgid where the object is found.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

objectstore_tool: lookup objects by name

If the object is not a parsable JSON string, assume an object name and
look it up in all the PGs. If multiple objects have the same name, only
apply the command to one of them. It is primarily useful in a test
environment where the names of the tests objects are known and only a
small number of objects exists. It replaces the following:

    path='--data-path dev/osd0 --journal-path dev/osd0.journal'
    for pgid in $(./ceph_objectstore_tool $path --op list-pgs) ; do
      object=$(./ceph_objectstore_tool $path --pgid $pgid --op list |
               grep '"oid":"NAME"')
      test -n "$object" && break
    done
    ./ceph_objectstore_tool $path --pgid $pgid "$object" remove

with:

    ./ceph_objectstore_tool $path NAME remove

http://tracker.ceph.com/issues/10192 Fixes: #10192

Signed-off-by: Loic Dachary <ldachary@redhat.com>

objectstore_tool: refactor list-lost and fix-lost

Abstract out the PG exploration loops and encapsulate the list-lost and
fix-lost semantic in a callable object.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

objectstore_tool: update usage strings

Signed-off-by: Loic Dachary <ldachary@redhat.com>

common/Initialize.h: add header file to Makefile.am

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #3081 from dachary/wip-10246-centos7

doc: add centos 7 to ceph-deploy installation

Reviewed by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3031 from dachary/wip-10199-config-get-xml

common: admin sock output XML elements whitespace

Reviewed-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@redhat.com>

Merge pull request #2959 from ceph/wip-10104

rados.py: fix misnamed 'wait_*' routines, add true wait/wait-with-cb

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2985 from CzBiX/quick-fix

rgw: check endpoints is not empty before use it.

This fixes a regression due to commit 3c4b782bcb5460523982a04b1e5c7963e08a1687

Reported-by: Mykola Golub <to.my.trociny@gmail.com>

pybind/test_rados: add test for wait_for_complete_and_cb()

Also, factor out some common code between the two tests

Signed-off-by: Dan Mick <dan.mick@redhat.com>

rados.py: fix misnamed 'wait_*' routines, add true wait/wait-with-cb

Also modify aio_read test for wait: write an object, take its active set
down, try to aio_read; verify read doesn't complete until active set is
allowed back up

Fixes: #10104
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #3076 from ceph/wip-mon-mdsdne

MDSMonitor.cc: fix assertion caused by MDSMap::STATE_DNE state

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3072 from ceph/wip-10231-gperftools-location

heap_profiler: support new gperftools header locations

gitbuilders are happy

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3084 from ceph/wip-rados-include

librados: Fix rados include problem reported by "sponce"

Reviewed-by: Sage Weil <sage@redhat.com>

librados: Fix rados include problem reported by "sponce"

Caused by a03f85a8e7fab296ea2df70a929a1c5e4aa0f7fb

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #3059 from dachary/wip-10125-radosgw-init

rgw: run radosgw as apache with systemd

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: run radosgw as apache with systemd

Same as sysv.

http://tracker.ceph.com/issues/10125 Fixes: #10125

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #2997 from ceph/wip-snapmapper

osd: require SNAPMAPPER feature; drop tolerate_collection_add_enoent

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2979 from ceph/wip-kill-category

osd: kill 'category' from pg_stats_t

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2694 from dachary/wip-9728-jerasure-neon

erasure-code: jerasure support for NEON

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3082 from scuttlemonkey/wip-doc-10247

Changed os-recommendation table sorting to remove implied bias.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3048 from ceph/wip-10209

osd: tolerate sessionless con in fast dispatch path

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3049 from dachary/wip-10211-erasure-code-buffer-alignement

erasure-code: enforce chunk size alignment

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/release-notes: v0.89

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

Changed os-recommendation table sorting to remove implied bias.

Fixes: #10247
signed-of by: Patrick McGarry <pmcgarry@redhat.com>

documentation: add centos 7 to ceph-deploy installation

http://tracker.ceph.com/issues/10246 Fixes: #10246

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3024 from ceph/wip-librados-symbols

librados: Only public API symbols from the shared library

Reviewed-by: Loic Dachary <ldachary@redhat.com>

MDSMonitor.cc: fix assertion caused by MDSMap::STATE_DNE state

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3028 from majianpeng/fix4

Add perf reset command.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3074 from ceph/wip-10229

osdc/Filer: use finisher to execute C_Probe and C_PurgeRange

osdc/Filer: use finisher to execute C_Probe and C_PurgeRange

Currently contexts C_Probe/C_PurgeRange are executed while holding
OSDSession::completion_lock. C_Probe and C_PurgeRange may call
Objecter::stat() and Objecter::remove() respectively, which acquire
Objecter::rwlock. This can cause deadlock because there is intermediate
dependency between Objecter::rwlock and OSDSession::completion_lock:

 Objecter::rwlock -> OSDSession::lock -> OSDSession::completion_lock

The fix is exexcute C_Probe/C_PurgeRange in finisher thread.

Fixes: #10229
Signed-off-by: Yan, Zheng <zyan@redhat.com>

heap_profiler: support new gperftools header locations

The google/ headers location has been deprecated as of gperftools 2.0.
As of gperftools 2.2rc, the google/ headers will now give deprecation
warnings, and they will probably disappear in a future gperftools
update.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

mon: fix signed/unsigned comparison warning

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

0.89

Merge pull request #3065 from ceph/wip-cephtool-exec

qa: fix cephtool tests not getting run

Reviewed-by: Loic Dachary <ldachary@redhat.com>

qa: fix teardown in cephtool's test_mon_mds

This was missing from 17b5fc9a but we didn't notice
because the test wasn't being run by the gitbuilders.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3044 from osynge/wip=suse-systemd

better systemd support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3067 from dachary/wip-cephtool

fix cephtools running only part of the tests + avoid killing wrong daemons

Reviewed-by: Sage Weil <sage@redhat.com>

common/perf_counters: Remove the duplicated code.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

test/perf_counters: Replace perfcounters_dump to perf dump.

The func of command  perfcounters_dump and 'perf dump' are the same .
But from the print 'ceph --admin-daemon help', it only print 'perf
dump'. So replace.
In order to keep consistent, still keep perfcounters_dump in code for
old user.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

test/perf_counters: Add test case for cmd 'perf reset'

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/perf_counters: Add 'perf reset all|perfcountername' for 'ceph
--admin-daemon'

Using this command, we can reset all perfcounters or reset on specified
perfcounter.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #3068 from ktdreyer/doc-os-recommendations-cleanup

doc: os recommendations cleanup

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: rm Fedora minor versions in os recommendations

Fedora doesn't have point releases, so it doesn't make sense to talk
about "Fedora 18.0" or "Fedora 19.0" - it's just "Fedora 18" or "Fedora
19".

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

os/ObjectStore: drop tolerate_collection_add_enoent

This is no longer needed and clutters the interface.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: require SNAPMAPPER feature from peers

This was introduced before cuttlefish.  We require users to upgrade first
to a newer release, so there is no need to support a mixed cluster with
such old code.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3061 from ktdreyer/rpm-ceph-disk-man-page

packaging: package ceph-disk(8)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

packaging: package ceph-disk(8)

The ceph-disk man page was added in
a450cab2b8148cb8a9b043d629feccf89e5aabac, but this was not added to the
RPM or DEB packaging. Add it here.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

doc: rm old releases in os recommendations page

Similar to the change in commit
4cdc9c793627eec13c65543c09c604ec27a4dd8a, remove the references to old
Ceph releases.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

doc: clarify "B" flag in os recommendations page

We don't exactly do continuous builds on all the platforms marked with
"B", but we have published binary RPMs for them. Adjust the "B"
footnote definition to reflect this.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

qa: cephtool/test.sh use regular strings to list tests

Using the array notation to list test is error prone and more
complicated to write.

It also fixes a bug : only the first test of each series (MON, OSD, MDS)
was run and the others were ignored.

Signed-off-by: Loic Dachary <loic@dachary.org>

tests: vstart_wrapper.sh must call vstart.sh with a list of daemons

The list of daemons is provided in the $CEPH_START environment variable
but vstart_wrapper.sh incorrectly uses the $VSTART_ARGS empty variable.

As a consequence vstart.sh calls init-ceph stop which kills daemons that
do not belong to the test.

http://tracker.ceph.com/issues/10215 Fixes: #10215

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: enforce chunk size alignment

Let say the ErasureCode::encode function is given a 4096 bytes
bufferlist made of a 1249 bytes bufferptr followed by a 2847 bytes
bufferptr, both properly starting on SIMD_ALIGN address. As a result the
second 2048 had to be reallocated when bufferlist::substr_of gets the
second 2048 buffer, the address starts at 799 bytes after the beginning
of the 2847 buffer ptr and is not SIMD_ALIGN'ed.

The ErasureCode::encode must enforce a size alignment based on the chunk
size in addition to the memory alignment required by SIMD operations,
using the bufferlist::rebuild_aligned_size_and_memory function instead of
bufferlist::rebuild_aligned.

http://tracker.ceph.com/issues/10211 Fixes: #10211

Signed-off-by: Loic Dachary <loic@dachary.org>

common: allow size alignment that is not a power of two

Do not assume the alignment is a power of two in the is_n_align_sized()
predicate. When used in the context of erasure code it is common
for chunks to not be powers of two.

Signed-off-by: Loic Dachary <loic@dachary.org>

common: add bufferlist::rebuild_aligned_size_and_memory

The function bufferlist::rebuild_aligned checks memory and size
alignment with the same variable. It is however useful to separate
memory alignment constraints from size alignment constraints. For
instance rebuild_aligned could be called to allocate an erasure coded
buffer where each 2048 bytes chunk needs to start on a memory address
aligned on 32 bytes.

Signed-off-by: Loic Dachary <loic@dachary.org>

doc: Reformatting on rbd replay doc.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added rbd-replay-many and restructured index.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Fixed index syntax.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3051 from ceph/wip-doc-rbd-replay

doc: Document RBD Replay

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3060 from ktdreyer/doc-os-recommendations-centos-7

doc: add CentOS 7 to recommended OSes

Reviewed-by: John Wilkins <jowilkin@redhat.com>

doc: add CentOS 7 to recommended OSes

CentOS 7 was missing from the list of recommended OSes for Firefly. Add
it to the list.

Reported-By: Frank Even <lists+ceph.com@elitists.org>
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #2993 from theanalyst/s3/keystone-date-fixes

rgw: check for timestamp for s3 keystone auth

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3058 from nilamdyuti/wip-doc-ceph-disk

doc: Adds man page for ceph disk in TOC.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Adds man page for ceph disk in TOC.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

Merge pull request #3056 from ceph/wip-9997-1

client: invalidate kernel dentries one by one

Reviewed-by: John Spray <john.spray@redhat.com>

client: invalidate kernel dentries one by one

Our trick to trim the whole kernel dentry tree does not work for 3.18+ kernel.
The fix is trim kernel dentries one by one.

Fixes: #9997
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3052 from ceph/wip-flush-journal-fix

mds: fix race of trimming log segments

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #2994 from dachary/wip-10175-compile-dependencies

script replacement for deps.{rpm,deb}.txt

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Thorsten Behrens <tbehrens@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@redhat.com>

mds: fix race of trimming log segments

trimming old log segments can start new log segments, make sure
MDLog::trim_all() not expire newly started log segments.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3041 from xiaoxichen/cleanup-noop

Cleanup noop in message

Reviewed-by: Sage Weil <sage@redhat.com>

doc: Document RBD Replay

Signed-off-by: Adam Crume <adamcrume@gmail.com>

osd: tolerate sessionless con in fast dispatch path

We can now get a session cleared from a Connection at any time.  Change
the assert to an if in ms_fast_dispatch to cope.  It's pretty rare, but it
can happen, especially with delay injection.  In particular, a racing
thread can call mark_down() on us.

Fixes: #10209
Backport: giant
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3008 from ceph/wip-10135

mon: OSDMonitor: allow adding tiers to FS pools

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3050 from dachary/wip-10213-const

msg: do not const methods that return a mutable pointer

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

msg: do not const methods that return a mutable pointer

Add remove misleading const on post_rx_buffer argument.

http://tracker.ceph.com/issues/10213 Fixes: #10213

Signed-off-by: Loic Dachary <loic@dachary.org>

osd/osd_types: drop category from object_info_t

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: drop category from copy-get

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: drop cateogry support from CREATE

Signed-off-by: Sage Weil <sage@redhat.com>

rados: drop optional 'create [category]' arg

Signed-off-by: Sage Weil <sage@redhat.com>

rados: change 'df' output now that categories are gone

Signed-off-by: Sage Weil <sage@redhat.com>