rgw: fix s3 list buckets

Fixes: #11451
Backport: hammer
Got broken in commit:7dd54fa3621c04c8ea5723fb1bc06b91d81a0c6c.
Resurrect the option to list unlimited number of buckets using the S3
api.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4420 from tchaikov/wip-man-fix-desc-in-name

man: fix the description in NAME section

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4409 from ceph/wip-11437

rgw: don't use end_marker for namespaced object listing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4408 from ceph/wip-11436

rgw: adjust return code if can't find upload

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4407 from ceph/wip-11435

rgw: fail if parts not specified on complete-multipart-upload

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4423 from ceph/wip-11370

librbd: better handling for duplicate flatten requests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: better handling for duplicate flatten requests

A proxied flatten request could be replayed, resulting in a
-EINVAL error code being generated on the second attempt. Filter
out that error if it is known the parent did exist before the
op started.

Fixes: #11370
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4372 from xiaoxichen/fix_kv_compact_on_mount

Fix kv compact on mount

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4418 from oritwas/wip-11438

rgw: force content-type header for swift account responses without body

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: force content-type header for swift account responses without body

This regression was introduced in commit 4cfeca03ade21861ab70ca759f94eb244c16cb39

Fixes: #11438
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

man: fix the description in NAME section

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4370 from guangyy/wip-ec-test-fix

qa : misc fixes to stabilize test-erasure-code suite

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4411 from hjwsm1989/wip-monstore-pool

mon: ceph-monstore-tool must do out_store.close() in "store-copy" command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4417 from osynge/wip-LimitNOFILE

Increase max files open limit for OSD daemon.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tools: ceph-monstore-tool must do out_store.close()

this change fixes the "store-copy" command.
Like the bug reported in http://tracker.ceph.com/issues/10093.

Signed-off-by: huangjun <hjwsm1989@gmail.com>

Increase max files open limit for OSD daemon.

Under heavy load the number of file descriptors opened
by the OSD can go beyond the 64K file limit. This patch
increases the default to 128K.

Signed-off-by: Owen Synge <osynge@suse.com>

Merge pull request #4257 from rzarzynski/wip-10683

rgw: rectify support for GET on Swift's account with "limit" parameter == 0

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't use end_marker for namespaced object listing

Fixes: #11437
Don't deal with end_marker if there's a namespace set for the object
listing. The cur_end_marker.empty() check is incorrect if no end_marker
is provided, but a namespace is set. Just avoid this issue entirely, as
it's not needed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adjust return code if can't find upload

Fixes: #11436
Return NoSuchUpload (instead of NoSuchKey).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fail if parts not specified on complete-multipart-upload

Fixes: #11435
A complete multipart upload should not succeed if parts were not
specified. Also, adjust return codes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4406 from ceph/wip-11433

librbd: always initialize perf counters

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: always initialize perf counters

If ImageCtx::init() fails, perf counters are still used when closing
the image. To prevent uninitialized use, it's easiest to always
initialize them before anything that can fail in ImageCtx::init().

This was moved in 6055df398557e4dbcabc66f463a0a126f9b5dc74, but it's
not necessary to delay perf counter initialization for any extra
configuration.

Fixes: #11433
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4302 from ceph/wip-11339

client: drop inode when rmdir request finishes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4282 from ceph/wip-3645

mds: rename snapshot support

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com

Merge pull request #4405 from ceph/wip-11380

cls_rbd: get_features needs to support legacy negative tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4402 from ceph/wip-rbd-unmap-by-spec

rbd unmap <image-name>

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cls_rbd: get_features needs to support legacy negative tests

During Ceph upgrade testing, older Ceph test suites assume that
get_features will return -ENOENT if provided a missing snapshot.
Support these negative tests until the older releases are no
longer supported.

Fixes: #11380
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3991 from rzarzynski/wip-11087

rgw: multiple fixes for copying Swift objects

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3863 from rzarzynski/wip-10971

rgw: multiple fixes for Content-Length header generation in Swift API

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: make rgw_read_user_buckets() backward compatible.

Fixes: #10683
Backport: hammer
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: rectify support for GET on Swift's account with limit == 0.

Fixes: #10683
Backport: hammer
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #4400 from ceph/update-organization-1

Update Haomai's organization

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3847 from trociny/wip-test_with_crushtool.master

mon: ceph osd setcrushmap: test_with_crushtool improvements

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4343 from ceph/wip-11356

client: fix uninline data funtion

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4223 from ceph/wip-11254

mds: make sure lock state not stay in XLOCK/XLOCKDONE indefinitely

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4218 from ceph/wip-11048

mds: persist completed_requests reliably

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4204 from ceph/wip-10368-mk3

mds: separate MDLog::safe_pos from journaler

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Update Haomai's organization

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

test/cli-integration/rbd: add unmap test

Add a test to exercise rbd unmap.  This was inspired by and is mostly
for rbd unmap <image-name> (aka spec), but rbd unmap <device> is tested
as well.  Any significant change to krbd.cc or corresponding rbd cli
should be tested in the following three configurations at least:

- kernel < 3.14
- kernel >= 3.14, rbd.single_major=N
- kernel >= 3.14, rbd.single_major=Y

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: allow unmapping by spec

Make use of krbd_unmap_by_spec() and allow unmapping by <image-name>
(i.e. pool/image@snap spec) with a bunch of careful adjustments to the
fragile command args/options parsing process to make --pool, --image
and --snap options work as expected for rbd unmap <image-name> case.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: add krbd_unmap_by_spec()

Make unmapping by (pool, image, snap) spec possible.  For specs mapped
multiple times unmap one device per krbd_unmap_by_spec() call.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: rename should_match_minor() to have_minor_attr()

Rename and clarify why simply checking kernel version doesn't work.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

client: drop inode when rmdir request finishes

Current client code relies on cap message to trim unlinked inode
from cache. This method is too sensitive to message ordering. If
client receives the cap message while directory inode still contains
dentries that are referenced by unsafe requests, the inode can't
get trimmed.

The fix is, when rmdir/rename request finishes, try trimming the
unlinked inode again.

Fixes: #11339
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #4380 from ceph/wip-nodebug-build

tests: separate check_PROGRAMS from TESTS

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tests: separate check_PROGRAMS from TESTS

Not all programs that should be built during 'make check' should
be executed as tests.  A new variable, check_TESTPROGRAMS, was
introduced to specify which programs should be executed as a test
during a 'make check'.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4311 from yuyuyu101/wip-async-fix-4

Wip async fix 4

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4358 from gerhardqux/gerhardqux-nokey-exit-1

ceph-authtool: exit(1) when printing a non-existing key

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4012 from majianpeng/cache-tier-fadvise

Cache tier fadvise

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3606 from wonzhq/hitset-insert

osd: avoid inserting an op into hit set multiple times

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4378 from dachary/wip-11359-tier-tests-disable

tests: comment out unstable tiering tests

tests: comment out unstable tiering tests

So that make check does not randomly fails.

http://tracker.ceph.com/issues/11359 Refs: #11359

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4377 from liewegas/wip-git.ceph.com

use git.ceph.com

Reviewed-by: Alfredo Deza <adeza@redhat.com>

use git://git.ceph.com

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-authtool: exit(1) when printing a non-existing key

When printing a non-existing key, ceph-authtool exits with a success
value:

[root@ip-172-31-3-178 ~]# ceph-authtool /etc/ceph/ceph.mon.keyring -p -n
client.doesntexist ; echo $?
entity client.doesntexist not found
0

Expected result: 1

Signed-off-by: Gerhard Muntingh <gerhard@qux.nl>

Merge pull request #4373 from dachary/wip-releases

releases: table of estimated end of life

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

releases: table of estimated end of life

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4339 from ceph/wip-da-SCA-20150317

Various SCA and Coverity fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4367 from ceph/wip-ceph-disk-default-cluster

ceph-disk: set '--cluster=ceph' in 'prepare' subcommand

Reviewed-by: Loic Dachary <ldachary@redhat.com>

os/LevelDBStore:fix bug when compact_on_mount

Initialize the performance counter(l_rocksdb_compact)
before use it in compact().

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/RocksDBStore: fix bug when compact_on_mount

Initialize the performance counter(l_rocksdb_compact)
before use it in compact().

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Merge pull request #4366 from BCLibCoop/bclibcoop/rgw-hostnames

rgw: Document & improve region hostnames

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

TestPGLog.cc: fix -Wsign-compare

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

TestErasureCodeShec.cc: fix -Wsign-compare

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test_async_driver.cc: fix UNINIT

Fix for:

CID 1254380 (#1 of 1): Uninitialized scalar variable (UNINIT)
 uninit_use_in_call: Using uninitialized value sa. Field
 sa.sin_zero is uninitialized when calling connect

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

psim.cc: change initialzation of VLAs

Use memset to fill integer VLAs.

Fix for:

CID 1219649 (#1 of 1): Uninitialized scalar variable (UNINIT)
 uninit_use: Using uninitialized value primary_count[primary]

CID 1219648 (#1 of 1): Uninitialized scalar variable (UNINIT)
 uninit_use: Using uninitialized value first_count[osds[0UL]]

CID 1219647 (#1 of 1): Uninitialized scalar variable (UNINIT)
 uninit_use: Using uninitialized value count[osds[i]]

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

systest_runnable.h: mark copy ctor as explicit

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

common/Thread.h: mark copy/move ctor as explicit

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

cls_rbd.cc: reduce scope of local variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ReplicatedBackend.cc: init 'Message *commit' to NULL

Init variable with NULL to make sure it crashes in error
case (would be a bug) if NDEBUG disable the assert() call.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/PG.h: mark copy ctor explicit

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

msg/Dispatcher.h: make copy/move ctor explicit

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test_shared_cache.cc: reduce scope of variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw_rest_user.cc: UserQuotas init vars in ctor init list

Fix for:

[src/rgw/rgw_rest_user.cc:631]: (performance) Variable 'bucket_quota' is
 assigned in constructor body. Consider performing initialization in
 initialization list.
[src/rgw/rgw_rest_user.cc:632]: (performance) Variable 'user_quota' is
 assigned in constructor body. Consider performing initialization in
 initialization list.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

client/Client.cc: fix uninit variables

This is caused by assert(). If assert is disabled we run into use
of uninitialized variables. Return -EINVAL in case assert() is
disabled.

Fix for:

[src/client/Client.cc:7791]: (error) Uninitialized variable: lock_state
[src/client/Client.cc:7829]: (error) Uninitialized variable: lock_type

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

libcephfs.h: fix unnecessary forward declarations

This is an issue in case non-cpp code. Move the declaration of
inodeno_t and vinodeno_t structs into __cplusplus block where
they are needed.

Fix for cppcheck issue:

[src/include/cephfs/libcephfs.h:89] -> [src/include/cephfs/libcephfs.h:74]:
 The struct 'inodeno_t' forward declaration is unnecessary. Type struct is
 already declared earlier.
[src/include/cephfs/libcephfs.h:93] -> [src/include/cephfs/libcephfs.h:83]:
 The struct 'vinodeno_t' forward declaration is unnecessary. Type struct is
 already declared earlier.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

os/FileStore.cc: fix format qualifier in COMMIT_SNAP_ITEM

All places COMMIT_SNAP_ITEM is used the variables are
'unsigned long long'. Change the format qualifier to reflect
this and to fix:

[src/os/FileStore.cc:1341]: (warning) %lld in format string (no. 1)
 requires 'long long *' but the argument type is 'unsigned long long *'.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/MDCache.cc: fix potential null pointer dereference

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/../chain_xattr.cc: fix suspicious usage of 'sizeof'

Use sizeof(char) instead of sizeof('\0').

Fix for:

[src/test/objectstore/chain_xattr.cc:164]: (warning) Suspicious
 usage of 'sizeof' with a numeric constant as parameter.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

TestErasureCodeShec: fix some -Wsign-compare

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw_rest_log.h: remove http_ret from some RGWOp_*_list classes

Remove http_ret from child class since there is already an integer
with the same name in the parent/base class.

Fix for:

[src/rgw/rgw_rest_log.h:79] -> [src/rgw/rgw_rest.h:275]: (warning) The
 class 'RGWOp_MDLog_List' defines member variable with name 'http_ret'
 also defined in its parent class 'RGWRESTOp'.
[src/rgw/rgw_rest_log.h:99] -> [src/rgw/rgw_rest.h:275]: (warning) The
 class 'RGWOp_MDLog_Info' defines member variable with name 'http_ret'
 also defined in its parent class 'RGWRESTOp'.
[src/rgw/rgw_rest_log.h:182] -> [src/rgw/rgw_rest.h:275]: (warning) The
 class 'RGWOp_DATALog_List' defines member variable with name 'http_ret'
 also defined in its parent class 'RGWRESTOp'.
[src/rgw/rgw_rest_log.h:202] -> [src/rgw/rgw_rest.h:275]: (warning) The
 class 'RGWOp_DATALog_Info' defines member variable with name 'http_ret'
 also defined in its parent class 'RGWRESTOp'.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

qa : misc fixes to stabilize test-erasure-code suite

Signed-off-by: Guang Yang <yguang@yahoo-inc.com>

ceph-disk: set '--cluster=ceph' in 'prepare' subcommand

otherwise the 'ceph' default value won't be catched by "main_prepare()",
this breaks TEST_corrupt_and_repair_replicated. seems the behaviour
was changed in python 2.7.9, so we move the default value to where
the '--cluster' argument is added instead of setting it in the
main argument parser.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: Changes in region hostname do not seem to take effect until radosgw is restarted.

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>

Merge pull request #4335 from liewegas/wip-osd-df

mon: fix min/max variance for 'cpeh osd df' when osds are marked out

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: Document a lurking problem I see with subdomain/domain splitting

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>

rgw: Speed up the building of valid de-duplicated hostnames for regions.

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>

doc: Document region hostnames settings.

The hostnames key in region settings was undocumented. Also includes how
the setting interacts with the ``rgw dns name`` configuration setting.

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>

Merge pull request #3481 from ktdreyer/wip-rpm-rm-old-fedoras

ceph.spec.in: remove reference to EOL Fedoras

Merge pull request #4316 from Ved-vampir/librdb

librdb: add perf counters descriptions

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4361 from oritwas/wip-11091

rgw: shouldn't return content-type: application/xml if content length is 0

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4351 from ceph/wip-11256

Wip 11256

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4363 from nilamdyuti/wip-doc-ceph-object-gateway

doc: Corrected the steps of Apache configuration for Debian-based
distros in object gateway installation guide.

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

doc: Corrected the steps of Apache configuration for Debian-based distros
in object gateway installation guide.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

mon: make note about osd df calc bug

Signed-off-by: Sage Weil <sage@redhat.com>

mon: fix min variance calc in 'osd df'

If the reweight is 0, ignore the osd for the purposes of calculating the
min.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4354 from zhouyuan/fix_ct_doc

doc: Fix misleading overlay settting in Cache Tier

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4359 from ceph/wip-fix-ceph-dencoder-build-master

Move ceph-dencoder build to client
Rework mds/Makefile.am to support a dencoder client build
rgw/Makefile.am: Populate DENCODER_SOURCES properly
Dencoder should never be built with tcmalloc

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

rgw: shouldn't return content-type: application/xml if content length is 0

Fixed: #11091

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Move ceph-dencoder build to client

The patch simply moves the ceph-dencoder build from server part of the
Makefiles to client part of the Makefiles.

Refs: #10691

Signed-off-by: Boris Ranto <branto@redhat.com>

Rework mds/Makefile.am to support a dencoder client build

The patch adds all the mds sources to DENCODER_SOURCES to allow a
dencoder client build. The patch also splits the Makefile.am file to
better accomodate the change.

Refs: #10691

Signed-off-by: Boris Ranto <branto@redhat.com>