rgw: implement Swift-specific authentication engines.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement RGWKeystoneAuthEngine.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: infrastructure for authentication engine-dependent ACL checks.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: share the ACL interpreter across multiple identity appliers.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: infrastructure for object-based Swift authentication.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: use RGWIdentityApplier across ACL implementation.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: enable management of admin privileges.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: decouple system user from admin user.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: split account owner from identity used for authorization.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: make use of RGWIdentityApplier as req_state::auth_identity.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: introduce RGWIdentityApplier for representing identities.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: remove get_keystone_{url,admin_token} from RGWSwift.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: switch from RGWSwift to KeystoneService admin_token/url.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: naturalize the transplanted get_keystone_{url,admin_token} in KeystoneService.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: ONLY move RGWSwift::get_keystone_{url,admin_token} to rgw_keystone.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: rework handling of rgw_keystone_accepted_[admin_]roles.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: cosmetic for RGWKeystoneTokenCache::RevokeThread.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: RevokeThread is now a part of RGWKeystoneTokenCache.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: ONLY move parts of KeystoneRevokeThread's code to rgw/rgw_keystone.cc

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement singleton pattern over RGWKeystoneTokenCache.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: replace RGWPostHTTPData with RGW[Keystone]HTTPTransceiver.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: improve encapsulation and const-correctness in the RGWHTTPClient family.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: ONLY move RGWPostHTTPData's definitions to rgw_http_client.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: ONLY move RGWPostHTTPData to rgw_http_client.h.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement RGWHTTPHeadersCollector.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: rgw_user can be now explicitly initialized with tenant and ID.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: improve const-correctness of rgw_open_cms_envelope.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: improve const-correctness of rgw_get_user_info_by_swift().

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: improve const-correctness of RGWHTTPArgs.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: improve const-correctness of KeystoneToken class.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: improve const-correctness of KeystoneToken.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: make rgw_get_user_info_by_uid to be const-correct.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #9037 from rzarzynski/wip-rgw-swift-quota

rgw: add support for account and container quotas of Swift API
Reviewed-by" Orit Wasserman <owasserm@redhat.com>

rgw: reuse populate_with_generic_attrs() across rgw_op.cc.Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: reuse populate_with_generic_attrs() across rgw_op.cc.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9367 from Yan-waller/yj-wip-rgwfixhextobuf

rgw/common: fix error return value of hex_to_buf.
Reviewed by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9438 from ceph/wip-cmake

cmake: rbd_mirror not linking rbd_test_mock

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9060 from mikulely/fix-rgw-multiste-doc

doc: fix rgw multiste config

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #9392 from yehudasa/wip-16072

rgw/rgw_common.cc: modify the end check in RGWHTTPArgs::sys_get

Tested-by: Orit Wasserman <owasserm@redhat.com>
http://pulpito.ceph.com/owasserm-2016-06-01_20:43:36-rgw:verify-wip-orit-testing---basic-mira/

cmake: rbd_mirror not linking rbd_test_mock

Added rbd_test_mock lib, also changed name of
journal_test to journal_test_mock to mimic
automake naming.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #9209 from yehudasa/wip-15939

rgw: aws4 fixes

verified

Merge pull request #9084 from dzafman/wip-dz-misc

Wip dz misc

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9426 from linuxbox2/cmake-mds

cmake: change libmds back to a static library

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: restore static linkage (libmds)

Required by ceph-mds.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #9385 from ceph/wip-cmake-kefu

cmake: more fixes

fixes make install workflow

Merge pull request #9228 from liewegas/wip-bluestore-write

os/bluestore: new write path (checksums and compression)

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9106 from SUSE/wip-15869

rpm: unconditionally set ceph user's primary group to ceph (SUSE)

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

cmake: install cython modules

* fix CYTHON_ADD_MODULE() macro. because python_add_module() offered by
  FindPythonLibs.cmake creates a target with name of ${name}, which conflicts
  with existing targets like "rbd" or "rados". so we can not reuse the
  name in ${name}.pyx. and instead, we should specify the target name
  explicitly.
* add distutils_install_cython_module() function to build and install
  cython modules.
* we can split build and install of cython module, but the install phase
  always tries to build the module. so keep it this way. will look at it
  later on.
* move the variables initializations into the Distutils.cmake module.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install compressor plugins into ${pkglibdir/compressor

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install erasure plugins into ${pkglibdir}/erasure-code

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove duplicated file from ceph-osd

objclass/class_api.cc is already included in libosd

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix dependencies on tracing headers

group the header dependencies by its tp .so, so the traced target can
depend on them by the name of ${name}-tp.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: libradosstriper's OUTPUT_NAME should be radosstriper

and s/libradosstriper/radosstriper/ otherwise the created .so
filename would be liblibradosstriper.so with the default prefix.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install the artifacts the packaging requires

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install init script to etc/init.d

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add the autoconf path variables back

* partially revert 7a602ec.
* the directory variables created by automake, like "prefix", "bindir",
  and "libdir", are used for generating configuration_file() for substitution,
  and they should have the same names with ones from autotools.
* also fix the ${pkglibdir}, it should be the ${libdir}/${PACKAGE}. so
  the plugins are not installed into ${libdir}, in which the installed shared
  objects are supposed to be shared with other applications.
* install shared libraries into ${CMAKE_INSTALL_LIBDIR} instead of
  ${prefix}/lib. this complies to what ceph.spec.in requires:
  ceph.spec.in expects the shared libraries to be installed into
  ${_libdir}, and ${_libdir} is /usr/lib64 on an amd64 machine.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install ceph-post-file

and related pubkey for sftp

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add ceph-brag

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: project name should be "ceph"

so it would be easier to figure out paths, also
CMAKE_INSTALL_DOCDIR is composed using the PROJECT_NAME also.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install ceph-{detect-init,disk}

add a cmake module named Distutils.cmake for setting up python modules
using setup.py.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add ceph_rgw_{json,multi}parser

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: compile and install ceph-bluefs-tool

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/ceph_test_xattr_bench/ceph_xattr_bench/

to match with automake and packager

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: rename ceph-psim to ceph_psim

to match with automake and packager

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix bash_completion install path

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fsck: check for dup overlay keys

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: use common helper to verify blobs and refs

The checks are the same (or should be--we had missed a few).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add FLAG_NOCACHE flag; do not cache unbuffered writes

Add a Buffer flag to mark that a buffer should not be cached once it is
stable.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not use buffered bdev in write path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not rely on bdev buffered reads in read path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: no buffered write in wal path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: populate buffer cache on read

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: keep intrusive_list of WRITING buffers

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: simple per-collection lru for buffers

Size these using a global config.  This is only a starting point--we'll
obviously have to rework this to share memory across collections.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use bufferptr for csum_data

encode/decode of vector<char> is not optimized.  Bufferptr is a more
natural type here anyway.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objecstore: Adds a test case for compression stuff verification (incomplete)

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Fixes configuration observation.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Cosmetic fixes in bluestore logging

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Enables cow for cloning at bluestore for store test

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Fixes Bnode serialization/deserialization and removes legacy Bnode::ref_map

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

ceph_test_objectstore: extends SimpleObjectTest with the case where write happens for neighboring csum blocks to verify for potential alignment issue

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Removes legacy block_size retrieval

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: use WriteContext and do_alloc_write for _do_write_small

Kill some mostly-duplicated code

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: consolidate WriteContext items into a write_item

Also include b_off in there.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: avoid unnecessary write_onode calls

_wctx_finish callers always write the onode; we only need to worry about
our changes to the bnode.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop unused _pad_* methods

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop unused _pad_zeros args

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix offset skew check

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: ~0x -> ~

e.g., 0x432da000~1000 instead of 0x432da000~0x1000

I think it's sufficiently clear that the value after ~ should have the same
base as the first bit, and it's easier to read.  And less text.

Signed-off-by: Sage Weil <sage@redhat.com>

compressor: Extends decompressor interface to be able to provide compressed data length.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: compress on write

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not partially deallocate compressed blobs

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_write_big: limit size of blobs based on compression mode

We may want to compress in smaller chunks based on hints/policy.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: track new compression config options

Class-wide Compressor, compression mode, and options.  For now these are
global, although later we'll do them per-Collection so they can be pool-
specific.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: add length to the compression_header_t

Snappy fails to decompress if there are extra zeros in the input buffer.
So, store the length explicitly in the header to avoid feeding them into
the decompressor.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix BufferSpace::read()

- we weren't reading from 'clean' buffers
- restructured loop a bit chasing another bug (but it ended up being
  in the caller)

Signed-off-by: Sage Weil <sage@redhat.com>

librados: add COMPRESSIBLE and INCOMPRESSIBLE alloc hints

Signed-off-by: Sage Weil <sage@redhat.com>

compressor: add a get_type() method to Compressor interface

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix _do_read cached vs read result assembly

We weren't handling the case of

 read block 0~300
 cache bloc 100~100

where the result is read(head) + cached + read(tail). Restructure the
loop to handle this.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix _do_read read out of buffer cache

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix up _set_csum helper

- make it thread-safe
- call during mount

Signed-off-by: Sage Weil <sage@redhat.com>