rgw: fix multipart min part size

As part of the fix for wip-7169 it turned out that we removed
min_part_size. Looking back, the original implementation was broken
anyway and didn't do anything. This fixes it and makes it configurable.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix multipart upload listing

Fixes: #7169
A separate fix has been created for dumpling.

Previously we read the entire list of parts, disregarding the actual
marker and the requested max parts. This fix refactors the way we read
the list of parts (doing it in parts, using marker).

Create new upload-id format that is used to identify uploads with sorted
omap entries. Make sure we're backward compatible and handle correctly
mixed-versions rgw uploads.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

test/Makefile.am: update test_cls_rgw* rules

Need to link with libcls_user_client.a

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

test_rgw_admin_opstate: get it compiled

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1137 from ceph/wip-7060

fix #7060 mon scrub problem due to osdmap copying

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #1105 from ceph/wip-7168

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

OSDMap: use deepish_copy_from in remove_redundant_temporaries

Better not to open-code it.  We do a bit of extra work here, but it's worth
it.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1080 from onlyjob/master

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

OSDMap: fix damaging input osdmap from remove_down_temps

The default copy constructor copies shared_ptrs do vectors that are then
modified by apply_incremental, which means that the const osdmap argument
isn't in fact const.  Fix this by doing a deep(ish) copy.

Fixes: #7060
Signed-off-by: Sage Weil <sage@inktank.com>

OSDMap: deepish_copy_from()

Make a deep(ish) copy of another OSDMap.  Unfortunatley we can't make the
compiler-generated copy operator/constructors private until c++11.  :(

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1149 from kdreyer-inktank/wip-7245-redhat-udev

packaging: apply udev hack rule to RHEL

Reviewed-by: Sage Weil <sage@inktank.com>
Reviewed-by: Derek Yarnell <derek@umiacs.umd.edu>

packaging: apply udev hack rule to RHEL

In the RPM spec file there is a test to deploy the uuid hack udev rules
for older udev operating systems. This includes CentOS and RHEL, but the
check currently only is for CentOS, causing RHEL clients to get a bogus
osd rules file.

Adjust the conditional to apply to RHEL as well as CentOS. (The %{rhel}
macro is defined in both platforms' redhat-rpm-config package.)

Fixes http://tracker.ceph.com/issues/7245

Signed-off-by: Ken Dreyer <ken.dreyer@inktank.com>

Merge branch 'wip-user-quota-2'

Conflicts:
src/common/config_opts.h
src/rgw/rgw_common.h
src/rgw/rgw_json_enc.cc
src/rgw/rgw_quota.cc
src/rgw/rgw_rados.cc
src/rgw/rgw_rados.h
src/rgw/rgw_user.cc
src/rgw/rgw_user.h

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1143 from dachary/wip-vstart-disk-usage

unittests: fail early when low on disk

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1003 from dachary/wip-handle-forward-unittest

mon: test for MForward messages

Reviewed-by: Joao Eduardo Luis <joao@inktank.com>

mon: test for MForward messages

Creates two mons, send a mon command to the peon and check that the
command is forwarded to the leader.

Add a test to check that the connection features of the client are
preserved when the message reaches the leader.

Signed-off-by: Loic Dachary <loic@dachary.org>

mon: shell test helpers to run MONs from sources

The intent is to make it more convenient to reproduce a specific mon
behavior and observe the result by grepping the logs. It can be handy
for bug diagnostic. The test could be included as a unit test to
be run on make check.

The setup function will prepare a directory and kill leftover from a
previous run. The teardown function cleans up on success. The run
function is expected to be provided by the calling script and can make
use of the run_mon function to mkfs + run a monitor.

Signed-off-by: Loic Dachary <loic@dachary.org>

unittests: fail early when low on disk

Scripts from qa that are run as unittests via test/vstart_wrapper.sh may
fail because the partition on which it runs is low on space ( 95% full
). When it happens the cause of the problem may be unclear because it
is likely to show only as a client not being able to reach the mon.

A test is added in test/vstart_wrapper.sh to verify the disk space usage
using the same method as the mon would and fail with a detailed error if
it is the case.

Signed-off-by: Loic Dachary <loic@dachary.org>

rgw: simplify a code path

remove some duplicate code.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

radosgw-admin: add usage for user stats [--sync-stats]

add missing usage

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

doc: Switched logging to true. Now required for sync agent.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

cls_user: add generate_test_instances() implementation

to all the encoded/decoded stuff

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw, cls_user: handle error cases related to response decoding

Certain operations weren't handling errors in decode, add some missing
logic.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: implement restful set user quota request

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: new restful api for retrieving user quota info

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: quota thread for full user stats sync

Get user stats up to date periodically. Add configurables for different
periods, whether we update idle users.
Make sure radosgw-admin does not start the quota threads.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: round bytes to the next 4k for quota use

Be more consistent as we round it like this in the objclass, so we need
the accounting to match.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: sync bucket stats thread

a new thread that periodically sync stats of recently modified buckets.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw, cls_user: fix bucket creation

There's a single op to create and update the user bucket info, however,
the cases differ a bit, as we only need to guard against ENOENT if we're
updating the info.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

radosgw-admin: dump header when requested for user stats

header also contains sync timestamps, so we should dump it.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

radosgw-admin: update stats sync completion

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_user: op to keep track of sync completion time

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_user: keep time info about stats update

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_user: header cleanup

put stats under a new struct

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: user quota should rely on cached info

user quota should rely on cached info if it says that we're at the
limit, since backend info might not be up to date.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: more quota fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: configure user quota also through region map

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: more rgw_quota fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: some quota fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: enable user quota

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

radosgw-admin: user quota interface

also add needed functionality in RGWUser

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: some fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: implement quota handlers for user stats

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: extend infrastructure to handle user stats read

Some helper functions that return data in required format for quota, add
async functions.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: move some code around, no actual changes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: refactor quota cache

bucket quota cache is going to be reused, add a new abstract layer.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: rename RGWBucketStats to RGWStorageStats

make it more generic, will be reused for user's stats.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: pass bucket owner all around

User quota operations require that we know who the actual user we do the
operation on is. Pass that info when creating new object and when
removing objects.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

radosgw-admin: bucket stats --sync-stats [--bucket=...]

Sync user stats, either for a single buckets, or for all user's buckets.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls/user: some fixes + handle old buckets case

If a bucket was created prior to the new accounting, we can't substract
the old values, as the header stats didn't account for them.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw, radosgw-admin, cls/user: retrieve user stats

add a new op to retrieve user header, new radosgw-admin command to stat
user.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls/user: a new op to retrieve user header

user header holds user total stats

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls/user: accounting functionality

Keep user total accounting in omap header, handle add/dec functionality.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: move bucket add / remove to new user objclass

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: replace user bucket listing with objclass call

instead of directly accessing the omap operation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: new user objclass

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1102 from ceph/wip-rgw-civetweb-2

Wip rgw civetweb 2

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1116 from jcsp/doc_toofull

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1135 from ceph/use-ceph-sharedptr

libc++: convert tr1 uses

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

libc++: convert tr1 uses

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

doc: Added domain pool, and changed zone configs to use domain tool so they don't clash.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Change zone domain to include "domain" so it is clear.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added space between {Query_String} and [E=HTTP ... to solve for 500 error.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #1134 from ceph/wip-backend-uninit-use

ReplicatedBackend: fix uninitialized use warning/bug

Reviewed-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: fix uninitialized use warning/bug

Signed-off-by: Greg Farnum <greg@inktank.com>

doc: Fixed omission of a comma in zone configuration.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

FileStore: perform LFNIndex lookup without holding fdcache lock

Fixes: #7207
Signed-off-by: Samuel Just <sam.just@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

Merge remote-tracking branch 'origin/next'

mongoose: git rm src/mongoose

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1132 from ceph/wip-erasure-rule

osd/OSDMap: do not create erasure rule by default

Reviewed-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

civetweb: md5.inl -> md5.h

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

civetweb: add include to makefile

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: modify the civetweb rule

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

civetweb: fix module uri

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

civetweb: update submodule

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: switch mongoose to civetweb

Fixes: #7054
mongoose is now GPL, using the civetweb fork instead.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1131 from ceph/use-unordered-map

osd: use ceph:: scoped hash_map

Reviewed-by: Sage Weil <sage@inktank.com>

osd/OSDMap: do not create erasure rule by default

If we do, we will require the v2 feature bit from clients.

We could only include feature bits for rules that are actually referenced
by pools, but for now making the user create the rule is simpler.  There is
no need to create this rule ahead of time.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: use ceph:: scoped hash_map

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #1064 from ceph/port/libc++

support libc++
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge branch 'wip-client-io-final'

Reviewed-by: Greg Farnum <greg@inktank.com>
Reviewed-by: David Zafman <david.zafman@inktank.com>

PGBackend: clarify rollback_stash name and method comments

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: drop messages from down peers

This overlaps with the existing old_peering_msg() mechanism
except in one case: pulls from a replica not in the acting
set.  If such a replica gets marked down, we may resend
pulls to another replica without causing a new interval
to start.  If we recieved, but didn't process, a push in
response to such a pull prior to processing the map marking
the peer down, we might process the push after having reset
the pull state for a different pull operation.  We can
avoid this by discarding ops from down peers.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: make STARTSYNC a noop

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG::calc_acting: consider newest_update_osd when choosing backfill peers

We must include newest_update_osd->second.log_tail when considering backfill
peers because in GetLog we will request logs back to the min last_update over
our acting_backfill set.  This will result in our log being extended as far
backwards as necessary to pick up any peers which can be log recovered by the
union of newest_update_osd's log and that of the chosen primary.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: take ReplicatedPGRef in BlessedContext

Signed-off-by: Samuel Just <sam.just@inktank.com>

RadosModel: RollBack dirties the object

Signed-off-by: Samuel Just <sam.just@inktank.com>

RadosModel: undirty with update_object_undirty

Otherwise it doesn't behave properly with snapshots since
update_object_version doesn't add another snapshot entry.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::trim_object: remove unused int r; declaration

Signed-off-by: Samuel Just <sam.just@inktank.com>

common/hobject: print shard as unsigned, not as char

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::_process_copy_chunk: use append

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: fill in mod_desc for hitset repops

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: use append for write and writefull

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: fill in mod_desc in finish_copyfrom

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::finish_copyfrom: check new_obs.exists, not obs.exists

If the transaction previously created the object, the stat update
will have already happened.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: handle removing the old object in finish_copy_op

do_osd_ops will need to either copy the old version out of the
way or simply delete it depending on mod_desc.  Thus, defer
handling filling that part in until we finish the copy op.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: EOPNOTSUPP on ec_pool if there are omaps in the COPY_GET

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: make COPY_GET async read friendly

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: requeue in_progress_async_reads on_change

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: support async reads on ec pools

Signed-off-by: Samuel Just <sam.just@inktank.com>