test/rados/misc.cc: add long locator key and namespace tests

Signed-off-by: Samuel Just <sjust@redhat.com>

LFNIndex: use chain_getxattr_buf

LFN attrs can be longer than these hard coded limits, let's just do
buffer doubling everywhere.

Signed-off-by: Samuel Just <sjust@redhat.com>

chain_xattr: add chain_getxattr_buf

Otherwise callers need to implement buffer doubling in a lot of cases,
which is error prone.

Signed-off-by: Samuel Just <sjust@redhat.com>

LFNIndex::lfn_parse_object_name: return int rather than bool

All current callers (except list_objects) already expect
ret < 0 for error.  Adjust list_objects to skip on EINVAL
instead of 0.

Signed-off-by: Samuel Just <sjust@redhat.com>

LFNIndex::list_objects: lfn_translate does not set errno

Signed-off-by: Samuel Just <sjust@redhat.com>

FileStore::set_xattr_limits_via_conf: add warning if max xattr size smaller than max name

Signed-off-by: Samuel Just <sjust@redhat.com>

test/objectstore/chain_xattr.cc: add test for ensure_single_attr

Signed-off-by: Samuel Just <sjust@redhat.com>

LFNIndex: ensure that lfn and directory attrs are written atomically

We leave skip_chain_cleanup = false since the object may have been
written using code that split the lfn attr over multiple attrs.

Signed-off-by: Samuel Just <sjust@redhat.com>

chain_[f]getxattr: always use size, no reaon to consider CHAIN_XATTR_MAX_BLOCK_LEN

Users of setxattr can now set attrs larger than that size.

Signed-off-by: Samuel Just <sjust@redhat.com>

chain_xattr: s/onechunk/skip_chain_cleanup, add ensure_single_attr

onechunk was a deceptive name since it didn't actually ensure that
a single chunk would be used.  Rename to ensure_single_attr.  Also,
add a parameter to ensure that we use a single attribute.  We need
these to be distinct since we have LFN attrs which have been split
over 254 byte xattrs since after hammer which we need to correctly
clean up.  However, we need to ensure going forward that those attrs
are never split over more than one chunk -- it's not atomic.
Further, skip_chain_cleanup should imply ensure_single_attr, so make
them template params and add a static check to make sure we don't
mess this up in the future.

Signed-off-by: Samuel Just <sjust@redhat.com>

os/,osd/: restructure the rados name length check

Enforce locator length vs the max name length and also
introduce a namespace length limit.

In addition to these checks, also pass the head object to the
ObjectStore implementation to validate.  This allows LFNIndex to account
for the idiosyncracies of its filename escaping and for different xattr
value max sizes.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #8378 from liewegas/wip-pgls-pgid

osdc/Objecter: use full pgid hash in PGNLS ops

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8069 from somnathr/wip-dyn-throttle-doc

Adding documentation on how to use new dynamic throttle scheme

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8429 from ErwanAliasr1/evelu-broken-cephtool-test-mon

tests: Fixing broken test/cephtool-test-mon.sh test

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #8411 from theanalyst/rgw/unused-var

rgw_admin: remove unused parent_period arg
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8426 from dillaman/wip-striper-logs

Striper: reduce assemble_result log level

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8421 from tchaikov/wip-noexcept

os/ObjectStore: add noexcept to ensure move ctor is used

Reviewed-by: Casey Bodley <cbodley@redhat.com>

tests: Fixing broken test/cephtool-test-mon.sh test

Since the merge of pr #7693, 'ceph command' to get the help is invalid.
As a result, 'test/cephtool-test-mon.sh' test was broken

This patch simply change the 'ceph command' by a 'ceph --help command'

Since this change the test is passing again.

Signed-off-by: Erwan Velu <erwan@redhat.com>

Merge pull request #8422 from ceph/wip-fix-cmake

CMake: For CMake version <= 2.8.11, use LINK_PRIVATE

Merge pull request #8410 from theanalyst/rgw/help-args

rgw: add a few more help options in admin interface

Merge pull request #8393 from jcsp/wip-15309-2

qa: update rest test cephfs calls (part 2)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8402 from ilc/master

configure: Add -D_LARGEFILE64_SOURCE to Linux build.

Merge pull request #8420 from dzafman/wip-15347

test: Fix test to run with btrfs which has snap_### dirs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8423 from wjin/fix

osdmap: rm nonused variable

Reviewed-by: Sage Weil <sage@redhat.com>

qa/workunits/rbd: qemu tests need to wait for image to be createdReviewe

qa/workunits/rbd: qemu tests need to wait for image to be created

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8427 from ghxandsky/master

set 128MB tcmalloc cache size by bytes

set 128MB tcmalloc cache size by bytes

Signed-off-by: Star Guo <star.guo@mevoco.com>

Merge pull request #8314 from dx9/wip-unittest-erasure-code-plugin

unittest_erasure_code_plugin: fix deadlock (Alpine)

Merge pull request #7693 from aclamk/auto_complete_python

ceph: bash auto complete for CLI based on mon command descriptions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8366 from yangdongsheng/trival

build: fix compiling warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Striper: reduce assemble_result log level

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: qemu tests need to wait for image to be created

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph_test_rados_api_tier: parse env

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_list: parse env

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: tolerate pgls commands with full hash

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: discard pgnls op that is outside pg bounds

This should only happen with a buggy client, but we should avoid crashing,
and send a polite error message back.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: use full hash value for pg[n]ls ops

Normal ops do this so they can behave when racing with split; pgnls ops
are no different.

In particular, this fixes a bug where we have an old OSDMap that doesn't
reflect a split, and the OSD replies with a 'next' value of the PG's new
max.  If we resend the same value to that PG, it'll be out of bounds,
and BlueStore will notice.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8380 from dillaman/wip-cls-fadvise

cls_rbd: pass WILLNEED fadvise flags during object map update

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

osdmap: rm nonused variable

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

CMake: For CMake version <= 2.8.11, use LINK_PRIVATE

Signed-off-by: Haomai Wang <haomai@xsky.com>

os/ObjectStore: add noexcept to ensure move ctor is used

otherwise vector::push_back() will use the copy ctor if it resizes,
to enforce its strong exception guarantee.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge branch 'wip-cycles'

common/Cycles: Do not initialize Cycles globally

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

common/Cycles: Do not initialize Cycles globally.

Move initialization to the few tests that actually use it.

Fixes: http://tracker.ceph.com/issues/15225
Signed-off-by: Richard W.M. Jones <rjones@redhat.com>

unittest_erasure_code_plugin: fix deadlock caused by locked mutex in cancelled thread

Added unlock when thread is cancelled.

Signed-off-by: John Coyle <dx9err@gmail.com>

Merge pull request #8193 from liewegas/wip-bluestore

os/bluestore: a few fixes

test: Fix test to run with btrfs which has snap_### dirs

Fixes: http://tracker.ceph.com/issues/15347
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #8404 from cbodley/wip-rgw-realm-pull-fix

radosgw-admin: fix for 'realm pull'
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8417 from dillaman/wip-librbd-mirroring-unsupported

librbd: avoid throwing error if mirroring is unsupported

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8325 from trociny/wip-rbd-mirror-test-misc

test: rbd-mirror: script improvements for manual testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8413 from runsisi/wip-fix-eexist

cls_journal: fix -EEXIST checking

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: avoid throwing error if mirroring is unsupported

Attempting to remove an image will remove the image from the mirroring
directory.  However, if the OSD is older and doesn't support this
new feature, avoid throwing an error.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: add exclusive flag to set_as_default()

this dodges the race in RGWRealm::create() and RGWZoneParams::create()
that decides whether to set the new object as a default. by calling
set_as_default() with exclusive=true, it will fail with EEXIST if a
default is already set

it also fixes an issue with 'realm pull' on a secondary zone, where a
'default' zone may be created but never actually set_as_default()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add exclusive flag to RGWRealm::create_control

this was causing failures in 'radosgw-admin realm pull':

  ERROR creating control for new realm realm0: (17) File exists

'realm pull' calls RGWRealm::create() with exclusive=false, but
RGWRealm::create_control() was using exclusive=true

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw_admin: improve the orphans find help

Adding  a few missing commandline switches and improve the help message

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #8406 from dmick/jewel

debian/rules: put init-ceph in /etc/init.d/ceph, not ceph-base

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8324 from tchaikov/wip-mon-cleanup

mon: remove remove_legacy_versions()

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

rgw: add a few missing cmdline switches in help

Add a few missing commandlien switches for zone
operations (zone_id,source_zone,read_only) to the help options

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

cls_journal: fix -EEXIST checking

Signed-off-by: runsisi <runsisi@zte.com.cn>

rgw_admin: remove unused parent_period arg

This variable seems to be unused, removing from help and code

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

debian/rules: put init-ceph in /etc/init.d/ceph, not ceph-base

When the package name changed from ceph to ceph-base, dh_installinit
started installing the init script into /etc/init.d/ceph-base.  Fix
this by using --name ceph with dh_installinit, which requires
1) naming the .init file ceph-base.ceph.init, and
2) calling dh_installinit separately for each package

Fixes: http://tracker.ceph.com/issues/15329
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #8371 from runsisi/wip-fix-eexist

cls_rbd: fix -EEXIST checking in cls::rbd::image_set

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

configure: Add -D_LARGEFILE64_SOURCE to Linux build.

Without -D_LARGEFILE64_SOURCE we can't build against libxfs,
because off64_t must be defined.

Signed-off-by: Ira Cooper <ira@redhat.com>

Merge pull request #8400 from qhjindev/fix_mon

mon: remove unnecessary comment for update_from_paxos

Merge pull request #8399 from oritwas/wip-cmake-mirrorwatcher

cmake: add missing librbd/MirrorWatcher.cc and librd/ObjectWatcher.cc

mon: remove unnecessary comment for update_from_paxos

The return value comment for update_from_paxos is unnecessary.

Signed-off-by: Qinghua Jin <qhjin_dev@163.com>

cmake: add missing librbd/MirrorWatcher.cc and librd/ObjectWatcher.cc

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8394 from xiexingguo/xxg-wip-compile

rgw: fix compiling error

Merge pull request #8358 from dachary/wip-releases

releases: what is merged where and when ?

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.cz>
Reviewed-by: Sage Weil <sage@redhat.com>

releases: what is merged where and when ?

Signed-off-by: Loic Dachary <loic@dachary.org>

releases: update understanding the release cycle

Signed-off-by: Loic Dachary <loic@dachary.org>

release-notes: v10.0.5 release notes

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #7875 from badone/wip-coverity-rgw_file-unused

[rgw] Check return code in RGWFileHandle::write

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix compiling error

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #8200 from H3C/wip-rgw-bugfixb

rgw/rgw_admin:fix bug about list and stats command
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

qa/workunits/rbd: use cluster1/2 instead local/remote as cluster names

After adding tests that start daemons in both clusters, local/remote
names look confusing.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

qa/workunits/rbd: add helper to determine rbd-mirror local cluster

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

qa: update rest test cephfs calls (part 2)

"mds stat" now gives fsmap output rather than
mdsmap.  Update the rest api test's expectations.

Fixes: http://tracker.ceph.com/issues/15309
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #8373 from wjwithagen/patch-4

ceph-detect-init/run-tox.sh: FreeBSD: No init detect

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #8387 from dreamhost/wip-mp-listpartresult-etag

rgw: Multipart ListPartsResult ETag quotes

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

qa/workunits/rbd: improvements for manual testing

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

bash_completion: now bash completion uses improved "ceph --completion"
Signed-off-by: Adam Kupczyk <a.kupczyk@mirantis.com>

tools: Auto complete feature for CLI.

Now logic moved from bash to python.
Not bind to bash yet. Use as 'ceph --comp osd ls'.
Able to fulfill commands and print command line help.
Signed-off-by: Adam Kupczyk <a.kupczyk@mirantis.com>

rgw: Multipart ListPartsResult ETag quotes

ListPartsResult output has always missed quotes on the ETag since it was
first committed.

Fixes: #15334
Backports: hammer, infernalis
Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

Merge pull request #8372 from jcsp/wip-15309

qa: update rest test cephfs calls

Merge pull request #8258 from asheplyakov/bug-12065

monclient: avoid key renew storm on clock skew

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8348 from xiexingguo/xxg-wip-failurepending

osd: fix failure report handling during ms_handle_connect()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8369 from xiexingguo/xxg-wip-tick

osd: fix tick relevant issues

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8354 from tchaikov/wip-missing-cls

osd/ClassHandler: only dlclose() the classes not missing

Reviewed-by: Sage Weil <sage@redhat.com>

os/filestore: fix a -Wunused-label warning in compiling.

os/filestore/FileStore.cc: In member function ‘int FileStore::_zero(const coll_t&, const ghobject_t&, uint64_t, size_t)’:
os/filestore/FileStore.cc:3328:2: warning: label ‘out’ defined but not used [-Wunused-label]
  out:
  ^

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

test/objectstore: fix a -Wsign-compare warning in compiling.

In file included from test/objectstore/store_test.cc:33:0:
../src/gmock/gtest/include/gtest/gtest.h: In instantiation of ‘testing::AssertionResult testing::internal::CmpHelperEQ(const char*, const char*, const T1&, const T2&) [with T1 = unsigned int; T2 = int]’:
../src/gmock/gtest/include/gtest/gtest.h:1524:23:   required from ‘static testing::AssertionResult testing::internal::EqHelper<true>::Compare(const char*, const char*, const T1&, const T2&, typename testing::internal::EnableIf<(! testing::internal::is_pointer<T2>::value)>::type*) [with T1 = unsigned int; T2 = int; typename testing::internal::EnableIf<(! testing::internal::is_pointer<T2>::value)>::type = void]’
test/objectstore/store_test.cc:1088:285:   required from here
../src/gmock/gtest/include/gtest/gtest.h:1448:16: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (expected == actual) {
                ^

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

test/system: fix a -Wsign-compare warning in compiling.

test/system/st_rados_create_pool.cc: In function ‘std::__cxx11::string get_temp_pool_name(const char*)’:
test/system/st_rados_create_pool.cc:128:9: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   assert(ret < sizeof(poolname));
         ^

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #8355 from dillaman/wip-15029

librbd: send notifications for mirroring status updates

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cls_rbd: fix -EEXIST checking in cls::rbd::image_set

Signed-off-by: runsisi <runsisi@hust.edu.cn>

librados_test_stub: add new cls_cxx_read2/write2 methods

Fixes: http://tracker.ceph.com/issues/15332
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: pass WILLNEED fadvise flag for object map update ops

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

objclass: add cls_cxx_read2/cls_cxx_write2

These new methods support passing op flags (e.g. cache hints).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8382 from athanatos/wip-15241

osd: fix epoch check in handle_pg_create

Reviewed-by: Sage Weil <sage@redhat.com>

OSD::handle_pg_create: check same_primary_since

Rather than add a flag to handle_pg_peering_evt, check
same_primary_since here and pass the create event to
handle_pg_peering_evt as if it originated at the current epoch (the
project_pg_history checks in handle_pg_peering_evt will be noops).

Fixes: tracker.ceph.com/issues/15241
Signed-off-by: Samuel Just <sjust@redhat.com>