cmake: add rgw_basic_types.cc to librgw.a

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #6784 from ceph/wip-5073

rgw multitenancy

Buckets and users can now have a tenant associated with them. Buckets can be specified as [<tenant>:]<bucket> and users can be specified as [<tenant>$]<uid>. The existing users and buckets are part of the default global tenant.
This is first phase. Future work will add tighter integration (e.g., swift tenant mapping, s3 subdomain tenant mapping, etc.)

Reviewed-by: Pete Zaitcev <zaitcev@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6727 from smithfarm/wip-index

doc/dev/index.rst: begin writing Contributing to Ceph

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/dev/index.rst: begin writing Contributing to Ceph

Replace existing index.rst with the beginnings of a comprehensive
developer guide.

Move the old index.rst to index-old.rst.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6775 from mkoderer/doc/fixowner

doc: remove unnecessary period in headline

Reviewed-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

doc: remove unnecessary period in headline

Signed-off-by: Marc Koderer <marc@koderer.com>

rgw: remove comments

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a minor cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't re-set bucket tenant and name when selecting location

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix a typo

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't allow cross-tenant bucket creation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: avoid calling rgw_make_bucket_entry_name() when not needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: objexp hint name backward compatibility

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: inherit bucket tenant from user if not specified

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6739 from dreamhost/wip-13920

Gentoo: _FORTIFY_SOURCE fix.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6771 from dillaman/wip-backtrace-fixes

common: re-enable backtrace support

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6657 from kylinstorage/wip-rbd-nbd

rbd-nbd: network block device (NBD) support for RBD

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6769 from aiicore/configure_fix

configure.ac: macro fix

Reviewed-by: Boris Ranto <branto@redhat.com>

configure.ac: macro fix

Configure fails with autoconf 2.63 on Centos 6.6 with:

./configure: line 34026: syntax error near unexpected token `newline'
./configure: line 34026: `  yes:no:'

Signed-off-by: Igor Podoski <igor.podoski@ts.fujitsu.com>

Merge pull request #5685 from ceph/wip-12133

cephfs-data-scan: scrub tag filtering (#12133 and #12145)

tools/cephfs: enable tag filter in DataScan

Fixes: #12133
Signed-off-by: John Spray <john.spray@redhat.com>

tools/cephfs: fix datascan kwarg parsing

Signed-off-by: John Spray <john.spray@redhat.com>

cls_cephfs: add PGLSCephFSFilter

Special purpose filter for use with CephFS disaster
recovery.

Fixes: #12145
Signed-off-by: John Spray <john.spray@redhat.com>

common: re-enable backtrace support

A recent porting effort disabled support for backtrace creation.  It
also didn't initialize all variables, which would lead to a crash when
deleting a backtrace instance.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6719 from jcsp/wip-13898

tools: Fix layout handing in cephfs-data-scan (#13898)

Update man page for new rbd nbd command

Signed-off-by: Yunchuan Wen <yunchuan.wen@kylin-cloud.com>

update test_librbd_fsx to support rbd-nbd

Signed-off-by: Yunchuan Wen <yunchuan.wen@kylin-cloud.com>

add rbd-nbd man page reference

Signed-off-by: Yunchuan Wen <yunchuan.wen@kylin-cloud.com>

add rbd-nbd wrap to rbd CLI

Signed-off-by: Yunchuan Wen <yunchuan.wen@kylin-cloud.com>

Merge pull request #6661 from xiaoxichen/rbd_misc_fix1

cls/cls_rbd.cc: fix misused metadata_name_from_key

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cls/cls_rbd.cc: no need to skip key == after.

We already skip the pair where key == after in both LevelDBWholeSpaceIteratorImpl::upper_bound
and RocksDBWholeSpaceIteratorImpl::upper_bound.

Fixes: #13922
Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Merge pull request #6742 from dillaman/wip-13913

librbd: automatically flush IO after blocking write operations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

init-ceph: fix systemd-run cant't start ceph daemon sometimes

Fixes: #13474
Signed-off-by: Chuanhong Wang <wang.chuanhong@zte.com.cn>

Merge pull request #6749 from dachary/wip-doc-install

doc: INSTALL redirect to online documentation

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tools/cephfs: fix layout handling in injection

When the stashed-layout handling stuff was added,
the guessing part got lost from the path where
we only had a single object.  Broke test_data_scan.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #6729 from majianpeng/librbd-journal

librbd: only send signal when AIO completions queue empty

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6751 from ceph/wip-cmake-strerror_r

cmake: define STRERROR_R_CHAR_P for GNU-specific strerror_r

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #5465 from yuyuyu101/wip-librbd-ap

librbd: support eventfd for AIO completion notifications

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cmake: define STRERROR_R_CHAR_P for GNU-specific strerror_r

This is what autotools does with AC_FUNC_STRERROR_R bit.  It takes us
from

    $ monmaptool --print nonexistant
    monmaptool: monmap file nonexistant
    monmaptool: couldn't open nonexistant: (2)

to

    $ monmaptool --print nonexistant
    monmaptool: monmap file nonexistant
    monmaptool: couldn't open nonexistant: (2) No such file or directory

on cmake - otherwise we end up using GNU-specific version as if it was
the POSIX version.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #6745 from drunkard/master

doc: fix typo in cephfs/quota

Merge remote-tracking branch 'gh/jewel'

doc: INSTALL redirect to online documentation

The INSTALL file has obsolete and misleading instructions to compile
from sources (-pg actually fails at link time). Replace with links to
the online installation and build from source documentations.

Signed-off-by: Loic Dachary <loic@dachary.org>

doc: typo fix in cephfs/quota

librbd/JournalReplay: Only Send signal when aio completions queue empty.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #6697 from XinzeChi/wip-scrub-test

test: fix osd-scrub-snaps.sh

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test: fix osd-scrub-snaps.sh

when we run run-make-check.sh in hdd disk, it would spend much more
time to finish scrub.

Signed-off-by: Xinze Chi <xinze@xsky.com>

test: add pg_scrub for ceph-helpers.sh

Signed-off-by: Xinze Chi <xinze@xsky.com>

librbd: Fix rebase with new io flow

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: fix lttng tracing argument mismatch

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: Add set_event_notify to AioImageRequestWQ

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: Remove unneeded set_event_notify

Since xlist clear method invoked below is enough to disassociate the xlist
item from the xlist, so the remove_myself() call in the destructor is safe
to invoke since it's already been removed from the list.

Signed-off-by: Haomai Wang <haomai@xsky.com>

Makefile: Add noinst headr files

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: Make rbd header file uses independent enum definition

Signed-off-by: Haomai Wang <haomai@xsky.com>

Librbd: Make AioCompletion complete doesn't unlock if callback

Signed-off-by: Haomai Wang <haomai@xsky.com>

Librbd: fix return code of EventSocket init and notify

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: Add ictx check to avoid AIO_TYPE_NONE completion

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: normalize notify return code

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: Fix incorrect api declaration

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: check event_notify to avoid extra logic

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

EventSocket: Add new event type pipe support

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

test: Add tests for getting arg of completion

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: Add interface to let user can get private data from comp

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

tests: Add tests for user io event notify

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: Add event notify interfaces

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: automatically flush IO after blocking write operations

This simplifies other state machines that previously had to flush IO
after blocking write operations.

Fixes: #13913
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6741 from SUSE/wip-tyepdef-typo-fix

include/rados/librados.h: fix typo

Reviewed-by: Loic Dachary <ldachary@redhat.com>

include/rados/librados.h: fix typo

This typo causes a warning when building the docs.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6740 from dillaman/wip-13911

librbd: ensure copy-on-read requests are complete prior to closing parent image

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: utilize common flush helper when closing parent images

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: only enqueue flush completion if a flush is pending

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Gentoo: _FORTIFY_SOURCE fix.

GCC under Gentoo has a patch that has a built-in define for
_FORTIFY_SOURCE if -O2 or higher are used. This causes warnings about
_FORTIFY_SOURCE redefined for every single object compiled.

Work around the issue by explicitly undefining _FORTIFY_SOURCE before we
define it again.

http://tracker.ceph.com/issues/13920 Fixes: #13920
Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

Merge pull request #6737 from dillaman/wip-13912

librbd: check for presence of journal before attempting to remove

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6668 from zaitcev/cleanup-dead-code-2

rgw: Remove unused code in PutMetadataAccount:execute

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #6738 from ceph/wip-rbd-cli-misc

rbd: fixes for refactored CLI and related tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6715 from jcsp/wip-librbd-cmake

cmake: update for recent librbd changes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6432 from ukernel/wip-11482

mds: fix client capabilities during reconnect (client.XXXX isn't responding to mclientcaps(revoke))

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #6736 from dillaman/wip-13914

librbd: not necessary to hold owner_lock while releasing snap id

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: check for presence of journal before attempting to remove

Fixes: #13912
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: helper method to determine if journal exists

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: not necessary to hold owner_lock while releasing snap id

Fixes: #13914
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: bail if too many arguments provided

The code has a catch clause for that, but it was being rendered useless
by the preceding

    if (command_spec.size() > matching_spec->size())
      positional_options.add(at::POSITIONAL_ARGUMENTS.c_str(), -1);

which names all (both expected and extraneous) positional arguments.

Change it to name only expected arguments, deriving the number of
expected arguments from the length of positional_opts vector, supplied
by each action.  This works for all actions except "feature enable" and
"feature disable" which are specified as multitoken, so keep on passing
in -1 for those.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: don't append an extra newline after some errors

Don't append an extra newline after program_options-generated errors,
like "unrecognised option" or "the argument for option is invalid".

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

tests: update unmap.t CLI test

Fixup the exit code - the old CLI tried to differentiate between CLI
errors and action errors by returning EXIT_FAILURE in the former case.
Also remove a test that relied on a special case check in the old CLI.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

cmake: librbd needs libjournal and libcls_journal_client

Commit 4719696cadd1 ("cmake: updates for refactored librbd IO path")
fixed file lists but missed the link dependency - librbd now needs
libjournal and libcls_journal_client.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #6706 from xiexingguo/xxg-wip-13711

rbd: output formatter may not be closed upon error

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6734 from accelazh/master

Correct typo 'restared' to 'restarted'

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Correct typo 'restared' to 'restarted'

There is a typo in document "Monitoring OSDs and PGs" as below

    An OSD was ``down``, was restared, and is now ``recovering``.

This patch corrects 'restared' to 'restarted'. The whole code base
is also searched to ensure there is no more 'restared'.

Signed-off-by: Yilong Zhao <accelazh@gmail.com>

Merge pull request #6733 from tchaikov/wip-fixcodingstyle

CodingStyle: fix broken URLs

Reviewed-by: Loic Dachary <ldachary@redhat.com>

CodingStyle: fix broken URLs

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6724 from dachary/wip-13592-test-auto-repair

tests: fix race condition testing auto scrub

Reviewed-by: Xinze Chi <xinze@xsky.com>

mon/PGMonitor: MAX AVAIL is 0 if some OSDs' weight is 0

In get_rule_avail(), even p->second is 0, it's possible to be used
as divisor and quotient is infinity, then is converted to an integer
which is negative value.
So we should check p->second value before calculation.

It fixes BUG #13840.

Signed-off-by: Chengyuan Li <chengyli@ebay.com>
(cherry picked from commit 18713e60edd1fe16ab571f7c83e6de026db483ca)

os: FileStore::_destroy_collection may hide the real mistake.

Signed-off-by: Ruifeng Yang <yangruifeng.09209@h3c.com>
(cherry picked from commit 9e9770ca87720781264d2e283739fc9e197706c9)

Fix mon routed_request_tids leak

Signed-off-by: Ning Yao <zay11022@gmail.com>
(cherry picked from commit ba3c64ca705590f833806300461f3b98de0e62f8)

pybind: decode empty string in conf_parse_argv() correctly

cretargs is a array of c_char_p, which means ctypes has already
converted it to python byte strings. decode_cstr() would misinterpret
the empty string as a NULL c_char_p(), and convert it to None by
accident, resulting in errors when running commands like
'ceph config-key put foo ""'.

Since this is the only place we use arrays of c_char_p, just decode
it directly in conf_parse_argv(). Tested with python 2 and 3.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit f76d5d6fe6a9f92b1ec17191f5504e2f61e0ff28)

ceph_test_keyvaluedb_iterators: Fix broken test

Introduced by #6312
Signed-off-by: Haomai Wang <haomai@xsky.com>
(cherry picked from commit ce0369444558b6308e1e0ffa49ae68faaac2bd1d)

tests: fix race condition testing auto scrub

When testing auto scrub, waiting 20 seconds for the scrub to complete is
sometimes not enough and creates false negatives.

Split wait_for_scrub out of the repair helper so that it can be used to
wait for the scrub to happen instead of using a timer.

The scrub timestamp is obtained after removing the object, therefore
there is a chance for the scrub to be finished already. But since auto
scrub is scheduled every 5 seconds, it will only make the test wait an
extra 5 seconds and not hang forever.

http://tracker.ceph.com/issues/13592

Signed-off-by: Xinze Chi <xinze@xsky.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

cmake: update for recent librbd changes

Recent merge moved lots of things.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #6503 from XinzeChi/wip-repop-worker

osd: defer decoding of MOSDRepOp/MOSDRepOpReply

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6597 from ceph/wip-fix-kvdb-test

tests: ceph_test_keyvaluedb_iterators: fix broken test

Reviewed-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #6543 from efirs/optimize-stringify-call

common: reduce CPU usage by making stringstream in stringify function thread local

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>