rgw: generate version ids that don't change when urldecoded

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: hash bucket shard by original object name, not transformed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use bucket shard for olh operations

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: api adjustments following a merge

Merged bucket sharding and object versioning.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'origin/master' into wip-rgw-versioning-4

Conflicts:
src/cls/rgw/cls_rgw_client.cc
src/cls/rgw/cls_rgw_client.h
src/rgw/rgw_bucket.cc
src/rgw/rgw_common.h
src/rgw/rgw_op.cc
src/rgw/rgw_rados.cc
src/rgw/rgw_rados.h

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3460 from athanatos/wip-10524

Wip 10524

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3461 from athanatos/wip-10535

ceph_test_objectstore: enable keyvaluestore experimental option

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3462 from athanatos/wip-10536

ReplicatedPG::_scrub: don't record digests for snapdirs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3459 from athanatos/wip-10603

ObjectStore::_update_op: treat CLONERANGE2 like CLONE

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3350 from liewegas/wip-10477

osd: populate blocked_by with peers we are trying to activate

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3457 from athanatos/wip-txn-append

Wip txn append

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3351 from liewegas/wip-10512

osd: fix blocked vs blocking rqeuest requeue order on interval change

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3363 from liewegas/wip-cache-fix

osd: fix ReplicatedPG::cancel_proxy_read_ops()

Reviewed-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Merge pull request #3365 from liewegas/wip-peering

osd: new activating pg state between peering and active

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3425 from vuhuong/wip-xio-fork

xio: Enable xio option to call fork init

Reviewed-by: Matt Benjamin <matt@cohortfs.com>

Merge pull request #3453 from FrankYu/master

Doc: Fix the extra blank space in doc/start/quick-rbd.rst

Merge pull request #3451 from xiaoxichen/fix_memstore_statfs

Fix memstore free space caculation

Reviewed-by: Sage Weil <sage@redhat.com>

Doc: Fix the extra blank space in doc/start/quick-rbd.rst

Signed-off-by: Frank Yu <flyxiaoyu@gmail.com>

Merge pull request #3450 from zhouyuan/rgw_doc_typo

doc: Fix a typo in radosgw-admin doc

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Fix memstore free space caculation

Originally if memstore run out of space, it will report a very
large positive number as free space. For example:

root@bigmem:~# rados df
pool name                 KB      objects       clones     degraded      unfound           rd        rd KB           wr        wr KB
rbd                 12366704        92240            0            0           0            0            0        92240     12366704
  total used        12375877        92240
  total avail   36028797009199167
  total space        2611076

st->f_bavail = st->f_blocks - used_bytes / st->f_bsize

This is due to used_bytes is an unsigned value, so compiler make the whole statement unsigned.

Fix it by adding explicit type cast,

st->f_bavail = long(st->f_blocks) - long(used_bytes / st->f_bsize)

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

doc: Fix a typo in radosgw-admin doc

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Revert "Revert "Merge remote-tracking branch 'origin/wip-bi-sharding-3' into next""

Following a merge of next to master, the feature got reverted (because
it was reverted on next). Undoing.

This reverts commit 6613358ddc5339c8e33c409387fd6044db0b6f26.

Merge pull request #3246 from ceph/wip-9780-9781

ceph-objectstore-tool import/export and related fixes

Reviewed-by: Samuel Just <sjust@redhat.com>

rgw: set default value for swift versioning extension

Otherwise we're going to crash if not found.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

PGBackend: fix and clarify be_select_auth_object

Previously, auth would end up containing every object without a
self-evident defect -- even if they did not match each other.  Instead
of filtering out the non-matching items there, be_compare_scrubmaps now
returns one valid object and be_compare_scrubmaps gathers the other
which match it.

We can be smarter by doing this in be_select_auth_object and selecting
the largest matching set, but for now this is simpler.

Fixes: 10524
Signed-off-by: Samuel Just <sjust@redhat.com>

rgw: fix bucket removal with data purge

Called by radosgw-admin user rm, need to set the versioning status
correctly.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge branch 'next'

ObjectStore::_update_op: treat CLONERANGE2 like CLONE

Fixes: #10603
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3251 from theanalyst/fix/rgw/s3-10062

wip: rgw: check keystone auth also for s3 post requests

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3381 from liewegas/wip-pg-stat-updates

osd: avoid spamming monitor with unchanged pg stats

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3393 from ceph/wip-osd-df

osd: ceph osd df [tree] command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3409 from trociny/wip-osd_pool_flags

osd: new pool safeguard flags: nodelete, nopgchange, nosizechange

Reviewed-by: Sage Weil <sage@redhat.com>

rbd: ensure aio_write buffer isn't invalidated during image import

The buffer provided to aio_write shouldn't be invalidated until
after aio_write has indicated that the operation has completed.

Fixes: #10590
Backport: giant
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: assign versioned_epoch on bucket listing response

For some reason this was missing.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3426 from jdurgin/wip-10592

qa: disable automatic locking for manual locking test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3259 from trociny/pg_dump_plain

mon: improvements for 'ceph pg dump' in plain format

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #3431 from XinzeChi/wip-compile-warning

osd: fix some compile warning

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3436 from ceph/wip-10579

qa: move fs quota to its own dir

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa: move fs quota to its own dir

This is because running fs/misc on a kclient
should not include quota (quota not supported
on kernel client).

Fixes: #10579
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3402 from ceph/wip-10388

#10388 PerfCounters for MDCache: strays and recovery

osd: fix some compile warning

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

Merge pull request #3433 from FrankYu/master

Doc: Fix the typo in doc/rbd/rados-rbd-cmds.rst

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Doc: Fix the typo in doc/rbd/rados-rbd-cmds.rst

Fix the typo in doc/rbd/rados-rbd-cmds.rst, also add explanation/example
when creating an image without specifying pool.

Signed-off-by: Frank Yu <flyxiaoyu@gmail.com>

Merge pull request #3427 from jdurgin/wip-cram

test: fix rbd cli tests for new feature bit

Reviewed-by: Sage Weil <sage@redhat.com>

test: fix rbd cli tests for new feature bit

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: disable automatic locking for manual locking test

Automatic locking hides the ESHUTDOWN from the caller, which is how
this test detects that blacklisting works.

Fixes: #10592
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

xio: Enable xio option to call fork init

Ceph fork when running as daemon. This fix call xio option to enforce
Accelio to explicitly call ibv_fork_init() to avoid cq overrun issue.

Signed-off-by: Vu Pham <vu@mellanox.com>

osd: revert d427ca35404a30e1f428859c3274e030f2f83ef6

d427ca35404a30e1f428859c3274e030f2f83ef6 reversed the order of localt
and op_t, while some logic depends on the order of localt and op_t.

This commit revert d427ca35404a30e1f428859c3274e030f2f83ef6.

Fix http://tracker.ceph.com/issues/10534.

Change-Id: I5adba561d10274e3d12f89418a755c39e5e5f086
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

osd: Transction::append SHOULD NOT modify other.op_bl

the other.op_bl SHOULD NOT be changes during append operation,
we use additional bufferlist to avoid this problem

Change-Id: I5d4a8c9ed3d0b926ca4bbdcdadf72cb4d5fde53c
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

Merge pull request #3408 from ceph/wip-10257

mon: PGMonitor: several stats output error fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3423 from yuyuyu101/rename-test-msgr

test_msgr: Rename unittest_msgr to ceph_test_msgr

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test_msgr: Rename unittest_msgr to ceph_test_msgr

Because ceph_test_msgr will run a lot of times than unittest expected,
move it into qa suites.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

mon: PGMonitor: skip zeroed osd stats on get_rule_avail()

Fixes: #10257
Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

mon: PGMonitor: available size 0 if no osds on pool's ruleset

get_rule_avail() may return < 0, which we were using blindly assuming it
would always return an unsigned value.  We would end up with weird
values if the ruleset had no osds.

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #3192 from gravitystorm/patch-1

Replace placeholder with 'gateway'

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Replace placeholder with 'gateway'

This section is where all the placeholders are replaced by example text, this one had been missed out.

Signed-off-by: Andy Allan <github@gravitystorm.co.uk>

Merge pull request #3414 from vuhuong/wip-cmake-fix

cmake: Fix CMakelist.txt to remove depleted files

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3419 from Nairolf21/wip-documentation

doc: Introduce the term MON at http://ceph.com/docs/master/rados/

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Introduce the term MON at http://ceph.com/docs/master/rados/

http://tracker.ceph.com/issues/10203 Fixes:#10203

Signed-off-by: Florian Coste <fcoste21@gmail.com>

Merge pull request #3421 from hazem92/wip-documentation

doc: mon should be listed before osd

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: mon should be listed before osd

When deploying a Ceph cluster, the mon must be run first.
In the list shown at http://ceph.com/docs/master/rados/configuration/
it would therefore be better to have mon listed before osd.

http://tracker.ceph.com/issues/10204 Fixes: #10204

Signed-off-by: Hazem <hazem.amara@telecom-bretagne.eu>

Merge pull request #3420 from Anols/wip-documentation

doc: missing the reference to "ceph-deploy-new"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3418 from ThomasCantin/wip-documentation

doc: Network Configuration Reference duplicate string #3418

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Doc : Documentation#10205

Missing the reference of "ceph-deploy-new".

https://tracker.ceph.com/issues/10205

Signed-off-by: AYARI <ayari_anis@live.fr>

doc bug 10206

Merge pull request #3164 from ceph/wip-librbd-rados-stub

librados test driver and librbd unit test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3411 from ceph/wip-10576

ceph_test_rados_api_misc: do not assert rbd feature match

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph_test_rados_api_misc: do not assert rbd feature match

This test fails on upgrades when we (or the server) have new
features.  Make it less fragile.

Fixes: #10576
Signed-off-by: Sage Weil <sage@redhat.com>

rgw: cors set attrs on bucket using appropriate call

Following rebase, we should call the appropriate function when setting
attrs on a bucket.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: switch index ops to new apis

following a rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: break if done when timing out pending olh ops

Entries are now sorted by timestamp (older to newer). Break if found
that entry has not timed out.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: start pending entries attr name by timestamp

This will make it (roughly) possible to get these sorted by time.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: user swap() instead of assignment operator

More efficient this way. A fix following code review.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix unitest compilation

Add the needed extra param for bilog_flags.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle 'null' instance when setting delete marker

Don't set 'null' instance for a delete marker. We use empty string for
this specific case.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix setting of versioned epoch when removing object instance

Beforehand if rgwx-versioned-epoch was passed, we ended up always
creating a delete marker. We don't do that now if rgwx-version-id is not
specified, which allows the system user to remove an object instance and
set the epoch appropriately.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add bucket index log flags, use these for marking versioned ops

We want to mark operations as versioned. This is needed by the sync
agent, so that we know how to handle bucket index log entries. We only
want to apply link-olh and unlink-instance operations when it's
versioned.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix use of rgw_remove_object() versioning_status param

Param passed is the versioning status, and not a boolean.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw, cls_rgw: log versioned operations in bucket index log

Add the following log ops:
 - link_olh
 - link_olh_del
 - unlink_instance

Log operations if needed. Use current version field in the log entries
to specify versioned epoch.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: cleanup, add read_bucket_header()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: fix rgw_bi_log_entry::dump()

Add missing instance field.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: bound bi log listing appropriately

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: only retry on ECANCELED before doing the operation

After we did the operation (e.g., link olh, remove) we don't need to
retry. We got ECANCELED, therefore the operation has already been
applied by another user.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix olh guarding scheme

is_olh() will return false if object is not yet an olh, although we
started the process into turning it into one. We should use the
existence of the olh tag instead.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: don't reuse -o command line option

Follow up on code review

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: multi object delete sends delete marker info

The api requires that we send information about whether the operation
was on a delete marker (either created or removed), and what was the
corresponding version id.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: delete obj, handle null instance in all paths

When passing rgw_obj structures to the bucket index, we should remove
the "null" instance, pass an empty instance instead.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: when deleting object version return -ENOENT if needed

If can't do bi_get() on the object, we should treat it as non existent
and just return.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: when removing delete marker, clean up instance entry

Since we create the instance entry at the bucket index, we should also
remove it. Otherwise we end up with objects that have bucket index
entries, but don't have olh. In these cases we'll be failin operations
on these objects, as their bucket index olh tag will always mismatch the
actual object olh tag.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: multi object delete supports versioning

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: break out of inifinite ECANCELED state

Don't wait forever, we might be in this state due to a bug, so it's
better to abort with EIO at this point.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw, cls_rgw: conditionally remove olh object

still not completely working, need to debug

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw, cls_rgw: clean up olh artifacts when dropping last version

This is still missing one piece, however, it works like this:
1. bucket index log reflects last version was removed (already did that)
2. olh entry is marked as 'pending removal'
 - any new versions coming in now to the olh entry will clear pending
   state
3. rgw conditionally removes olh object (still missing condition check)
 - condition checks should verify olh tag & version, and no pending
   modifications
4. if (2) is successful, rgw sends 'clear olh' request to bucket index
5. bucket index, if still 'pending removal' clears olh and plain entry

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: dump versioned epoch on system user bucket listing

The sync agent will need this info.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: only set olh attr on actual olh object

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: no need to set tag on object we're removing

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: only maintain object if there are preserved xattrs

rgw.obj_remove op doesn't need to keep the object around if there aren't
preserved xattrs found.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: limit print length of bufferlist buffer

These specific buffers are not null terminated.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: multipart meta object shouldn't be versioned

This object is already unique due to the upload id associated with it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove clutter

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>