rgw: Add documentation for RGW default quota

Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

rgw: Do not bother to check against conf values during quota handling

Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

rgw: Let the default quota settings take effect during user creation

Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

Merge pull request #9778 from ifed01/wip-bluestore-blob-reduce

os/bluestore: reduce bluestore blob

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9750 from ifed01/wip-bluestore-csum-output-fix

os/bluestore: fix improper blob's csum visualization.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9756 from liewegas/wip-bluestore-diet

os/bluestore: remove unused bluestore fields

Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

Merge pull request #9119 from runsisi/wip-fix-cmake-desc

cmake: update description for cmake module Findkeyutils.cmake

Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9799 from stiopaa1/common_address_helper_changeScope

common/address_helper.cc:move variables closer...

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9797 from Yan-waller/yj-wip-rgwcompiling

rgw: kill compiling warning

Reviewed-by: Casey Bodley <cbodley@redhat.com>

common/address_helper.cc:move variables closer...

... to the point where they are first used.

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

rgw: kill compiling warning

test/rgw/test_http_manager.cc: In member function ‘virtual void HTTPManager_SignalThread_Test::TestBody()’:
test/rgw/test_http_manager.cc:40:23: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   for (int i = 0; i < num_requests; i++) {
                       ^

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #9764 from vasukulkarni/master

mention change of owner for /var/log/ceph

Merge pull request #9599 from sdieffen/feature15904

added -f flag for formatting to rados command line tool

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9785 from cbodley/wip-rgw-httpmgr-select

rgw: fix deadlock in RGWHTTPManager when HAVE_CURL_MULTI_WAIT=0
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9765 from dx9/wip-run-cmake

cmake: add arguments to cmake invocation to support cache configuration

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #9634 from ceph/cmake-git-rev

Update git rev parsing in CMake

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #9601 from ceph/wip-cmake

cmake: add ceph_test_* used by qa units and ceph-qa-suite tasks

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #9725 from SUSE/wip-16327

rgw: check for -ERR_NOT_MODIFIED in rgw_rest_s3.cc
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: add pipe fd to set for select() in do_curl_wait()

when HAVE_CURL_MULTI_WAIT is 0, the pipe fd is never added to the
readfds for select(), so FD_ISSET() is always false. this prevents us
from ever trying to read from the fd, and the pipe's buffer eventually
fills up and deadlocks callers of RGWHTTPManager::signal_thread() when
they try to write to the pipe

Fixes: http://tracker.ceph.com/issues/16368
Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add unit test for HTTPManager

unit test to detect a deadlock in RGWHTTPManager::signal_thread() when
HAVE_CURL_MULTI_WAIT=0

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9779 from smanjara/wip-zone-rename

rgw: added zone rename to radosgw_admin

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9686 from weiqiaomiao/wqm-wip-rgw-output

rgw: remove unnecessary output
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9687 from theanalyst/rgw/orphans-find-msg

rgw-admin: don't print on ENOENT for orphans find
Reviewed-by" Orit Wasserman <owasserm@redhat.com>

Merge pull request #9736 from tchaikov/wip-test-timeout

test: disable dup test for mds respawn cmd

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9086 from ceph/wip-15849

msg/async/AsyncConnection: create writable event for in progress connection

Merge pull request #9461 from liewegas/wip-msgr2

doc: msgr2 protocol spec

Merge pull request #8963 from yuyuyu101/wip-15758

msg/async: Separated nonfast message and fast dispatch message thread

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: change bluestore_extent_ref_map_t key value(i.e. offset) from 64 to 32 bits.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: remove bluestore_blob_t::length field

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

rgw: added zone rename to radosgw_admin

Added zone rename option to radosgw-admin help

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #9759 from dillaman/wip-16352

rbd-mirror: ensure replay status formatter has completed before stopping

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9730 from dillaman/wip-16329

rbd-mirror: gracefully handle missing sync point snapshots

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9751 from dillaman/wip-16348

librbd: unable to disable journaling feature if in unexpected mirror state

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #8749 from Aran85/wip-check-failure-grace

mon/osdmonitor: decouple adjust_heartbeat_grace and min_down_reporters

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8827 from majianpeng/ceph-tell-osd

osd/OSD: make 'ceph tell osd.x get_command_descriptions' work.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9056 from stiopaa1/osd_pistate_localstruct

osd/OSD.cc: make pistate a local struct

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: disable dup test for mds respawn cmd

we set the CEPH_CLI_TEST_DUP_COMMAND enn var to verify the successful
commands are idempotent. but some of them are just not. among the other
things:
- ceph tell mds.a exit
- ceph tell mds.a respawn
the respawn command restart the mds daemon, its bind port changes and
all run-time status are reset. so strictly speaking, even the from the
point of view of client, this command is not idempotent. further more,
it fails the test, if the client sends the 2nd command too soon. because
the monitor might not able to update the re-spawned mds address before
the client asking for the new fsmap. so the cephfs client will just
use the old address of the specified mds, and hence will send the
request to port no one is listening anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add arguments to cmake invocation to support cache configuration

Signed-off-by: John Coyle <dx9err@gmail.com>

Also mention change of owner for /var/log/ceph

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

os/bluestore/bluestore_types: drop overlay types

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: drop overlay fields from wal and onode

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop overlay support

We can resurrect this from the history if/when we decide we want it.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: remove flags from lextent_t

Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror: ensure replay status formatter has completed before stopping

Fixes: http://tracker.ceph.com/issues/16352
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: drop old bluestore_extent_t

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: extent_t -> pextent_t

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9753 from liewegas/wip-readme

README: update with vstart, test, and make check instructions

Reviewed-by: Ali Maredia <amaredia@redhat.com>

README.md: note where make check test log output goes

Signed-off-by: Sage Weil <sage@redhat.com>

README.md: document running a test cluster

Signed-off-by: Sage Weil <sage@redhat.com>

README.md: simplify cmake instructions

Assume build is subdir of the ceph.git checkout.

Signed-off-by: Sage Weil <sage@redhat.com>

README.md: drop obsolete comment about debian squeeze

Signed-off-by: Sage Weil <sage@redhat.com>

README.md: get source first (including submodules!)

Signed-off-by: Sage Weil <sage@redhat.com>

README.md: document how to run unit tests (with autotools)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9151 from fullerdj/wip-djf-15706

rbd: add methods to set and get snapshot limits

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9292 from marcan/pyrbd-aio

pybind: AIO bindings for RBD

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9333 from VictorDenisov/consistency_groups_only_crud

rbd: initial operations for consistency groups

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9535 from xinxinsh/wip-bug

librbd: object_may_exist always return true when you write an empty object

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9682 from yuyuyu101/wip-fix-task-finisher

librbd: cancel all tasks should wait until finisher is done

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: fix improper blob's csum visualization.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

journal: do not log watch errors against deleted journal

The peer rbd-mirror process will eventually notice that the journal has
been removed and clean itself up. This avoids flooding the log with
expected error messages when journaling is disabled.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9567 from liewegas/wip-bluestore-blobcache

os/bluestore: convert cache to be blob-based (vs onode-based)

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: include device offsets in csum error message

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove Onode's BufferSpace

(now unused)

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: change _do_read to use blob cache

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9748 from Yan-waller/yj-wip-rgwstoreuserinfo

rgw/user: remove needless bufferlist encoding

build verfiied, there can be no side effect

librbd: force-remove journal when disabling feature and removing image

If mirroring is enabled, it's nearly certain that the peer rbd-mirror
daemon will be a registered client within the journal. Without
forcing the removal, this would prevent the removal from occurring.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ignore ENOENT error when removing image from mirror directory

Fixes: http://tracker.ceph.com/issues/16348
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9720 from dillaman/wip-16202

qa/workunits: rados cppool now requires --yes-i-really-really-mean-it

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

rgw/user: remove needless bufferlist encoding

info encoded but never be used.

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

mailmap: Victor Denisov affiliation

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: Add cg create, cg list, cg remove commands

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: Add cg_create, cg_list, cg_remove

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

cls_rbd: Add cg_create, cg_dir_add, cg_dir_remove, cg_dir_list

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

qa/workunits/rbd: increase writes in large image count test

This will help to test edge cases where the remote image does
or does not own the exclusive lock when the sync starts.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9708 from tchaikov/wip-make_rocksdb_makefile-without-rdb

script/make_rocksdb_makefile.sh: do not include rocksdb/tools/rdb/*

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Sage Weil <sweil@redhat.com>

Merge pull request #9713 from xiexingguo/xxg-wip-server_droppin

server: drop locks and auth pins before waiting for trunc

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9710 from xiexingguo/xxg-wip-server_duplock

Server: avoid duplicated call to acquire_locks()

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9706 from xiexingguo/xxg-wip-server_dcr

mds/server: avoid side-effect of more() method

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9647 from xiexingguo/xxg-wip-server_hcr

server: fix potential access violation

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8737 from david-z/wip-mds-slow-req-output

mds: ceph status outputs mds slow request for better monitoring

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

rbd-mirror: prune sync points referencing missing snapshots

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: ignore empty snapshot sequence mapping

This invalid condition will be gracefully detected and handled
when the snapshot mappings are computed.

Fixes: http://tracker.ceph.com/issues/16329
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: inline and simplify _read_whole_blob

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use _dump_blob_map helper

Unify bnode and onode dump output.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do writes into blob buffer cache

No change to the read-side yet.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: use single bl_eq helper

Use a single bl comparator helper that dumps buffers if they do not match.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: clean up csum read error checks

Fix warning message and return error code.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use Blob* in WriteContext::write_item

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: switch to intrusive_set for blobs

Switch from a map of bare bluestore_blob_t's to an intrusive_set of
Blob structs that includes id, bluestore_blob_t, and BufferSpace.  The
buffer cache portion isn't used yet, but it is properly initialized with
the Cache parent.

There is no real functional change here except that during clone we move
the existing Blob from the Onode's BlobMap to the Bnode's without any
copying.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: move blob_map out of onode_t

Move the blob_map index out of the onode_t proper.  None of the (important)
onode_t methods use the blob_map, which is expected since blobs may exist
in the onode or bnode map.  Instead, move it into Onode, parallel the
Bnode counterpart.

This will allow us to create a blob map that includes the buffer cache
with an encode/decode and lifecycle independent of the onode_t.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9526 from liewegas/wip-bluestore-csum2

os/bluestore: vary csum chunk sizes based on hints

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: add crc32c_16 and crc32c_8

This is much faster than a slice-by-8 crc16, perhaps even without the
intel instructions.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

compressor/snappy: style cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

compressor/snappy: compress to page-aligned memory buffer

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove unused ext_offset region_t field

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: simplify spare read path, remove pextent constraint

The previous read code had the constraint that a physical extent had to
be a multiple of the csum chunk size.  This isn't needed: we might have
a csum_block of 1MB and min_alloc_size of 4KB and that's okay.

Collapse the two helpers into a single loop that uses the blob_t::map()
method to do the pextent part of the read.  This is simpler and avoids
the temporary extents2read structure.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: csum_block -> csum_chunk

Be consistent with terminology.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: ensure blob_t::put_ref respects csum chunk size

We can't deallocate part of a csum chunk.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: test larger writes from Synthetic

Signed-off-by: Sage Weil <sage@redhat.com>