Merge pull request #2588 from xinxinsh/wip-9595

Fix #9595

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2628 from ceph/wip-client-flock

Wip client flock

Add support for file locking to the userspace client, and improve blocked-lock cancellation so that it doesn't remove locks that succeeded when racing.

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #2732 from dachary/wip-9526-crush-rename-bucket

ceph osd crush rename-bucket srcname dstname

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2731 from dachary/wip-9790-display-auid

auth: add display of auid to auth get / list when it is not the default

Reviewed-by: Sage Weil <sage@redhat.com>

mon: unit tests for osd crush rename-bucket

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

mon: add the osd crush rename-bucket command

The synopsis is:

 osd crush rename-bucket name1 name2

It is made idempotent by interpreting -EALREADY as returned by
CrushWrapper::rename_bucket return as success.

The crush_rename_bucket method first checks for errors with
CrushWrapper::can_rename_bucket if there is no pending crush so that it
can return early and avoid the creation of a pending crush map.

If renaming is possible, CrushWrapper::rename_bucket is called on the
pending crush map (and creates it indirectly if it does not already
exists).

http://tracker.ceph.com/issues/9526 Fixes: #9526

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

crush: unit tests for CrushWrapper rename item/bucket methods

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

crush: add CrushWrapper::rename_bucket and can_rename_bucket

Rename a bucket and return -ENOTDIR if trying to rename an item. The
behavior is otherwise the same as rename_item and can_rename_item.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

crush: add CrushWrapper::rename_item and can_rename_item

The can_rename_item is a const method checking if renaming an item could
succeed. If not it returns a unique -errno code and a human readable
message message.

Trying to rename a non existent item into an existent item returns
-EALREADY which can be treated as success if renaming is to be
idempotent.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

add unittest for Striper::get_num_objects

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

crush: improve constness of CrushWrapper methods

A number of CrushWrapper get methods or predicates were not const
because they need to maintain transparently the rmaps. Make the rmaps
mutable and update the constness of the methods to match what the caller
would expect.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

when non-default striping is used, internal methods can operate on extra objects

Fixes: #9595
Signed-off-by: xinxin shu <xinxin.shu@intel.com>

Merge pull request #2712 from lpabon/bug-8766

Update vstart to setup users for s3-tests

Merge pull request #2705 from xinxinsh/master

return value of handle_message for MSG_OSD_SUBOP/MSG_OSD_SUBOPREPLY shou...

Reviewed-by: Sage Weil <sage@redhat.com>

auth: unit tests for auid display

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

auth: add display auid to KeyServer::encode_secrets

http://tracker.ceph.com/issues/9790 Fixes: #9790

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

auth: add display auid to KeyRing::encode_formatted

http://tracker.ceph.com/issues/9790 Fixes: #9790

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2726 from CzBiX/wip-doc-radosgw-federated-config

doc: correct command of `config push`

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

doc: correct command of `config push`

Signed-off-by: VRan Liu <gliuwr@gmail.com>

qa/workunits/fs/misc: Add a workunit for file lock interruption

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: fix neighbor lock check

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #2721 from wyllys66/wip-8942

librados: osd: Fixed JSON output for stray OSDs

Backport: giant, firefly
Reviewed-by: Sage Weil <sage@redhat.com>

doc/dev/sepia: notes about sepia email list and irc channels

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: v0.80.7

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2704 from almightybeeij/master

libcephfs.h libcephfs.cc : Defined error codes for the mount function

Reviewed-by: Greg Farnum <greg@inktank.com>

doc: update lab notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2720 from ceph/wip-5977

librbdpy: Added missing method docstrings

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

librbdpy: Added missing method docstrings

Several methods were missing docstrings, preventing the methods
from appearing in the generated documentation.  Ensured all methods
now have appropriate docstrings.

Fixes: 5977
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mds: reply -EOPNOTSUPP for unknown lock type

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #2718 from henrycc/wip-henrycc-inotable

mds: fix inotable initialization/reset

mds: fix inotable initialization/reset

interval_set::insert takes arguments start and len, not end.

Signed-off-by: Henry C Chang <henry.cy.chang@gmail.com>

Merge pull request #2714 from ceph/wip-9730

doc: `ceph-deploy mon create-initial` doesn't take arguments

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #2706 from dachary/wip-9747-ceph-spec

rpm: 95-ceph-osd-alt.rules is not needed for centos7 / rhel7

Backport: giant, firefly
Reviewed-by: Sage Weil <sage@redhat.com>

doc: remove node requirement from 'mon create-initial'

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

doc: remove whitespace

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

Update vstart to setup users for s3-tests

vstart.sh now creates the users for the default configuration
for the s3-tests, available on https://github.com/ceph/s3-tests.

Also updated the documentation to show the correct RadosGW port.

Signed-off-by: Luis Pabón <lpabon@redhat.com>

Merge pull request #2591 from majianpeng/fix2

os/FileStore: using FIEMAP_FLAGS_SYNC instead of fsync() before call fiemap

Backport: giant, firefly
Reviewed-by: Sage Weil <sage@redhat.com>

client: use finisher to abort MDS request

When a request is interrupted, libfuse first locks an internal mutex,
then calls the interrupt callback. libfuse need to lock the same mutex
when unregistering interrupt callback. We unregister interrupt callback
while client_lock is locked, so we can't acquiring the client_lock in
the interrupt callback.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: use atomic variable to track reference of MetaRequeset

this allow us to increase reference count of MetaRequest while not holding
the client_lock

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: allow interrupting blocked file lock operation

This commit introduce two new types of setfilelock request. Unlike
setfilelock (UNLOCK) request, these two new types of setfilelock request
do not drop locks that have alread been acquired, they only interrupt
blocked setfilelock request.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: register callback for fuse interrupt

libfuse allows program to reigster a callback for interrupt. When a file
system operation is interrupted, the fuse kernel driver sends interupt
request to libfuse. libfuse calls the interrupt callback when receiving
interrupt request.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

FileStore:Round offset of fiemap down aligned with CEPH_PAGE_SIZE.

There is a bug on xfs about fiemap. If offset unsigned, the result of
fiemap will leak some data.
Kernel commit eedf32bfcace7d8e20cc66757d74fc68f3439ff7 fix this bug.
To avoid this bug on kernel which don't apply this commit, in ceph we
make the offset down aligned with CEPH_PAGE_SIZE.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

rpm: 95-ceph-osd-alt.rules is not needed for centos7 / rhel7

The || instead of && had it always installed. That was fixed in EPEL
already.

http://tracker.ceph.com/issues/9747 Fixes: #9747

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

libcephfs.h libcephfs.cc : Defined error codes for the mount function
Used new error codes from libcephfs.h to replace the magic numbers in the mount functon found in libcephfs.cc.

Signed-off-by: BJ Lougee <almightybeeij@gmail.com>

Merge pull request #2701 from ceph/wip-9716

librados: Fix function prototypes in librados.h

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2678 from ceph/wip-8983

rados: Parse command-line arguments strictly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Reviewed-by: Loic Dachary <loic-201408@dachary.org>

return value of handle_message for MSG_OSD_SUBOP/MSG_OSD_SUBOPREPLY should be true

if the return value is true , it will skip unnecessary checks

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

Merge pull request #2683 from ceph/wip-7796

rgw: set length for keystone token validation request

Reviewed-by: Sage Weil <sage@redhat.com>

rados: Use strict_strtoll instead of strtoll

Signed-off-by: Adam Crume <adamcrume@gmail.com>

Merge pull request #2672 from ceph/wip-9513

Fix read performance regression in ObjectCacher

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

librados: Fix function prototypes in librados.h

Fixes: 9716
Signed-off-by: Adam Crume <adamcrume@gmail.com>

rados: Add tests for CLI strict parsing

Fixes: 8983
Signed-off-by: Adam Crume <adamcrume@gmail.com>

test: Fix rmpool in test_rados_tool.sh

Signed-off-by: Adam Crume <adamcrume@gmail.com>

Merge pull request #2675 from ceph/wip-coverity-20141003

Fix issues from SCA and Coverity

Reviewed-by: Sage Weil <sage@redhat.com>

client: add helper function that updates lock state

Signed-off-by: Yan, Zheng <zyan@redhat.com>

fuse: enable fuse_multithreaded by default

GETFILELOCK MDS request may block for a long time, so we need to
use multithread event loop,

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: posix file lock support

Signed-off-by: Yan, Zheng <zyan@redhat.com>

common: link mds/flock.o to libcommon

later commit will use this code to track file locks held by cephfs
client.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #2685 from hejiantao5/patch-1

Check the pointer before deleting

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2688 from leseb/fix-warning-message-img-format

Fix error message when stripping with format 1

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

Fixed JSON output for stray OSDs

Fix error message when stripping with format 1

Since the option '--format' for specifying the rbd image format is
deprecated, we should recommend '--image-format' instead.

Signed-off-by: Sébastien Han <sebastien.han@enovance.com>

Check pointer before deleting

Here I think should check the pointer, to avoid potential risk

Signed-off-by: Jiantao He <hejiantao5@gmail.com>

ceph_erasure_code_benchmark: fix parameter handling

Make sure k and m paramter are valid to prevent crash. Fix typo.

Fix for the following CID and other possible invalid combinations
of k/m parameter:

CID 1219466 (#1 of 1): Division or modulo by zero (DIVIDE_BY_ZERO)
 divide_by_zero: In expression rand() % (k + m), modulo by expression
 k + m which may be zero has undefined behavior.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #2686 from dachary/wip-client-dout

client: add missing dendl and s/dout/ldout/

Reviewed-by: John Spray <john.spray@redhat.com>

client: add missing dendl and s/dout/ldout/

Introduced in ce4436c9d4206ff4a4af7fdc16e571dcc77cd8b4

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2671 from ceph/wip-da-fix-make_check

Makefile.am: make sure everything was build before make check

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2669 from ceph/wip-9692

qa/workunits/fs/misc: fix syntax error

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2676 from ceph/wip-print-mode

client: print out mode, uid, gid if they are changed

qa/workunits/fs/misc: fix syntax error

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #2677 from ceph/wip-mds-coverity

Wip mds coverity

rgw: set length for keystone token validation request

Fixes: #7796
Backport: giany, firefly
Need to set content length to this request, as the server might not
handle a chunked request (even though we don't send anything).

Tested-by: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rados: Parse command-line arguments strictly

Many integers were parsed with functions that silently ignored junk at
the end of the string.  Extra characters now cause errors.

Fixes: 8983
Signed-off-by: Adam Crume <adamcrume@gmail.com>

mds: MDirUpdate: initialize discover count to 0 if we're not discovering

Otherwise it's left uninitialized and could do who-knows-what!

CID 1244229:  Uninitialized scalar field  (UNINIT_CTOR)
/messages/MDirUpdate.h: 55 in MDirUpdate::MDirUpdate(mds_rank_t, dirfrag_t,
int, std::set<int, std::less<int>, std::allocator<int>> &, filepath &, bool)()

Signed-off-by: Greg Farnum <greg@inktank.com>

mds: MDSAuthCaps: init "read" param in default constructor

CID 1244228:  Uninitialized scalar field  (UNINIT_CTOR)
/mds/MDSAuthCaps.h: 29 in MDSCapSpec::MDSCapSpec()()

Signed-off-by: Greg Farnum <greg@inktank.com>

client: print out mode, uid, gid if they are changed

These values can be printed by MClientRequest message output, but if
they're flushed out via cap updates we don't even get that. These
are important values that generally don't change often; so print
them out in debug logging.

Signed-off-by: Greg Farnum <greg@inktank.com>

test_rgw_admin_log.cc: refactor to use calloc()

Refactor code to use calloc() instead of malloc() to make
sure the char* TEST_BUCKET_OBJECT_SIZE memory is set to
zero before call put_bucket_obj().

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test_rgw_admin_log.cc: fix use-after-free

Fix for:

CID 1054876 (#1 of 1): Use after free (USE_AFTER_FREE)
 pass_freed_arg: Passing freed pointer bucket_obj as an
 argument to put_bucket_obj

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/librados/c_write_operations.cc: free ressources

Release completion as soon as no longer needed.

Fix for:

CID 1219593 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable completion going out of scope
 leaks the storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #2511 from dachary/wip-catch-options-errors

test,tools: catch option parsing exceptions

Reviewed-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@redhat.com>

os/FileStore: Because do_fiemap will do fsync, so don't do fsync()
before calling do_fiemap.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #2629 from dachary/wip-heap-profiler

perfglue: minors heap profiler updates and documentation

Reviewed-by: Jianpeng Ma <majianpeng@gmail.com>

rados_list_parallel.cc: cleanup before return from run()

Fix for another case of:

CID 717112 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable io_ctx going out of scope leaks
 the storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/librbd/fsx.c: fix potential unterminated string handling

In case of strncpy() make sure string is '\0' terminated. Before
usage of strcat() check if the destination has enough free space
to concat the char and to terminate the string to prevent illegal
memory access.

Fix for:

CID 1219459 (2 of 2): Buffer not null terminated (BUFFER_SIZE_WARNING)
 buffer_size_warning: Calling strncpy with a maximum size argument of
 1024 bytes on destination array goodfile of size 1024 bytes might
 leave the destination string unterminated.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ErasureCodeIsa.cc: reduce scope of variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

SyntheticClient.cc: reduce scope of variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/flock.cc: reduce scope of some variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/ReplicatedBackend.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/OSDMap.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mount.ceph.c: ensure '\0' terminated string

Reserve last char in array for '\0' to ensure termination
of the string.

Fix for:

CID 1128383 (#1 of 1): Buffer not null terminated (BUFFER_SIZE_WARNING)
 buffer_size_warning: Calling strncpy with a maximum size argument
 of 1000 bytes on destination array secret of size 1000 bytes might
 leave the destination string unterminated.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #2586 from ceph/wip-mon-status

mon: make health portion of ceph -s readable (multiple lines)

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #2673 from dachary/wip-ceph-erasure-code-corpus

build: add ceph-erasure-code-corpus to gitmodule_mirrors

Reviewed-by: Sandon Van Ness <sandon@inktank.com>

build: add ceph-erasure-code-corpus to gitmodule_mirrors

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Makefile.am: make sure everything was build before make check

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge remote-tracking branch 'upstream/giant'

Merge pull request #2616 from guangyy/wip-giant-9614

PG::actingset should be used when checking the number of acting OSDs for a given PG.

Backport: firefly
Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2663 from ceph/wip-9496

mon: PGMonitor: populate scrub timestamps with 'now' on pg creation

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2650 from ceph/wip-9128

Add reset_tp_timeout in long loop in add_source_info for suicide timeout

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2543 from ceph/wip-9419

Wip 9419

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2670 from dachary/wip-9700-cephtool-mon-osd

qa: move mon_mds tests last

Reviewed-by: John Spray <john.spray@redhat.com>