test/cls_rgw: fix test

Test needed to be updated, now using meta.accounted_size for object
size.

Fixes: #10707
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'upstream/next'

Merge pull request #3588 from athanatos/wip-10693

Wip 10693

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3587 from athanatos/wip-10633

ReplicatedPG::C_ProxyRead: fix dropped lock

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3538 from athanatos/wip-7862

osd: allow recovery below min size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3539 from athanatos/wip-7861

osd: allow writes on degraded objects

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'origin/next'

Conflicts:
src/rgw/rgw_op.cc
src/rgw/rgw_rados.cc
src/rgw/rgw_rados.h

Merge pull request #3578 from ceph/wip-10709

rbd: watch command should unwatch before exiting

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3583 from jdurgin/wip-qemu-iotests

qa: use correct binary path on rpm-based systems

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3584 from ceph/wip-optional-encoding

encoding: ignore uninitialized instantiation in boost::optional decode

Reviewed-by: Sage Weil <sage@redhat.com>

qa: use correct binary path on rpm-based systems

Fixes: #10715
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

rbd: watch command should unwatch before exiting

Switched the rbd CLI tool to use the new watch/unwatch API
and properly unwatch the image before exiting. Additionally,
the watch command opened the image twice -- once R/W and another
R/O.  This resulted in two watches being created for the image.
Cleaned up old/new format detection by using the existing API
methods.

Fixes: #10709
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

encoding: ignore uninitialized instantiation in boost::optional decode

POD types won't be initialized until the decode writes to them.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3562 from liewegas/wip-lttng

do_autogen.sh: default to --with-lttng, -L to build without

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3485 from jdurgin/wip-merge-diff-testing

Improve merge-diff test

do_autogen.sh: default to --with-lttng, -L to build without

Get developers building with LTTNG by default.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3549 from Ved-vampir/master

Ceph: Fix

Reviewed-by: Sage Weil <sage@redhat.com>

osd:  change pg_stat plain to display CRUSH_ITEM_NONE in pgmap output section.

Fixes: #10519
Signed-off-by: Kiseleva Alyona <akiselyova@mirantis.com>

Merge pull request #3501 from yuyuyu101/wip-fix-async

AsyncMessenger: Fix mix auth assert failed

Merge branch 'wip-rbd-listwatchers'

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

man: add rbd status to doc/man/8/rbd.rst

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>
Reviewed-by: Li Wang <liwang@ubuntukylin.com>

Merge pull request #3533 from ceph/wip-disaster-doc

doc: add cephfs disaster recovery guidance

Reviewed-by: Greg Farnum <gfarnum@redhat.com

doc: add cephfs disaster recovery guidance

This is a place to put some useful notes about
the new offline recovery tooling.

Signed-off-by: John Spray <john.spray@redhat.com>

test: add rbd status to src/test/cli/rbd/help.t

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>
Reviewed-by: Li Wang <liwang@ubuntukylin.com>

rbd: add status command to list the watchers of an image

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>
Reviewed-by: Li Wang <liwang@ubuntukylin.com>

Merge branch 'librbd-cache-iohint' of git://github.com/majianpeng/ceph

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/librbd/ImageCtx.cc
src/librbd/ImageCtx.h
src/osdc/ObjectCacher.cc
src/osdc/ObjectCacher.h

Merge pull request #3556 from trociny/image_watcher-fixup

librbd: RemoteAsyncRequest: fix comparison operator

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3537 from ceph/wip-10688-boost-157

#10688: support Boost 1.57.0

Reviewed-by: Loic Dachary <ldachary@redhat.com>

librbd: RemoteAsyncRequest: fix comparison operator

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd: use IO hints for read/write operations

Import, export, and benchmark operations now use the librados
IO hints to indicate the read/write pattern and whether caching
is required.

Fixes: #10462
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'origin/wip-4087'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/librbd/LibrbdWriteback.cc

Merge pull request #3545 from thesues/fix-cmp-xattr

Fix ReplicatedPG do_xattr_cmp_u64

Backport: giant, firefly
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3555 from ceph/wip-10106

rgw: flush xml header on get acl request

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3527 from wonzhq/op-stat

osd: don't log op stats for proxy read in the cache tier

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3503 from ceph/wip-pg-reqids

osd: index PGLog with optional extra_reqids in each log entry

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Wang, Zhiqiang <zhiqiang.wang@intel.com>

Merge pull request #3554 from ceph/wip-10572

rgw: finalize perfcounters after shutting down storage

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3323 from ceph/wip-10445

rgw: use gc for multipart abort

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph_test_rados: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3452 from majianpeng/iohint

Iohint in librbd cache.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: flush xml header on get acl request

Fixes: #10106
Backport: firefly, giant

dump_start() updates the formatter with the appropriate prefix, however,
we never flushed the formatter.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: finalize perfcounters after shutting down storage

Fixes: #10572
Backport: giant, firefly
First disable the storage subsystem, then disable perfcounters.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: Don't do readahead for random read.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3516 from ceph/wip-mdsmon-args

mon/MDSMonitor: fix gid/rank/state parsing

Reviewed-by: Huang Jun <hjwsm1989@gmail.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #3551 from ceph/wip-10698

rgw: fail s3 POST auth if keystone not configured

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ReplicatedPG::C_ProxyRead: fix dropped lock

Fixes: 10633
Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: only populate_obc_watchers if active

Signed-off-by: Samuel Just <sjust@redhat.com>

OSD: allow recovery in peered

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: go "peered" instead of "active" when < min_size

In the case of a replicated pool, the pg will transition to "peered"
rather than "active", where it can perform backfill bringing itself up
to min_size peers.  To that end, most operations now require
is_peered rather than is_active (OSDOps being the primary exception).
Also, rather than using the query_epoch on the activation message as the
activation epoch (for last_epoch_started) on the replica, we instead
use the last_epoch_started in the info sent by the primary.  This
allows the primary to not advance last_epoch_started past the last
known actual activation.  This will prevent later peering epochs from
requiring the last_update from a peered epoch to go active (might be
divergent).

Fixes: #7862
Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: move most of all_activated_and_commited into the AllReplicasActivated handler

It's confusing to have that logic split up so much.

Signed-off-by: Samuel Just <sjust@redhat.com>

PG::proc_master_log: assume les and history.les from log source

Otherwise, we might later go peered (not active) and not distribute
a non-0 last_epoch_started.  This should be safe as the log recipient
will have a last_update reflecting that interval as well.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd_types: add last_peered, last_became_peered

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: break waiting_for_peered out of waiting_for_active

waiting_for_peered now holds ops until peering completes (activation,
not necessarily state active).  waiting_for_active now holds
specifically MOSDOp blocked on:
- scrub
- replay
- state active

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: remove very obsolete comment about backfill_pos

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: add state PG_STATE_PEERED

We need a state to represent the pg state where we are below
min_size, but trying to recover.

Signed-off-by: Samuel Just <sam.just@inktank.com>

doc: add documentation for undersized

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #3553 from ceph/wip-10701

rgw: use strict_strtoll() for content length

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ReplicatedPG: block writes on degraded objects for ec pools

Signed-off-by: Samuel Just <sjust@redhat.com>

append_log: use remove_snap_mapped_object

Backport: giant, firefly
Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG::agent_maybe_evict: take write lock, skip on failure

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: if we got ENOENT and an obc, we need to check the locks

Otherwise, we might return ENOENT out of order.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG::get_rw_locks: use excl lock for read & write_ordered

Signed-off-by: Samuel Just <sjust@redhat.com>

osd_types: add excl obc lock

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG::cancel_pull: drop_recovery_read lock

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG::wait_for_unreadable_object: use prep_object_replica_pushes instead of recover_missing if not missing

Signed-off-by: Samuel Just <sjust@redhat.com>

PG::purge_strays: purge peer_missing as well

Backport: giant, firefly
Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: don't block writes on degraded objects

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: update peer_missing and local missing if we write a degraded object

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: always take recovery_read_lock for recovery

Signed-off-by: Samuel Just <sjust@redhat.com>

rgw: use strict_strtoll() for content length

instead of strict_strtol().

Backport: giant, firefly
Fixes: #10701
Reported-by: Axel Dunkel <ad@dunkel.de>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3476 from ktdreyer/wip-rpm-radosgw-logrotate

ceph.spec.in: move rgw logrotate to rgw subpackage

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3437 from diurchenko/patch-2

rgw: fix doc, wrong HTTP header name

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3546 from zhouyuan/204_post_container

rgw: Response 204 when post on containers

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fail s3 POST auth if keystone not configured

Fixes: #10698
This fixes issue introduced in 8b3dfc9472022ea45ad24e02e0aa21dfdad798f8,
where if user does not exist, we try keystone authentication. However,
if keystone is not configured we justt fall through without failing.
This would have failed later on due to bucket permissions, unless bucket
had a public write permissions.

Reported-by: Valery Tschopp <valery.tschopp@switch.ch>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

doc: Fixed hyperlink.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

rgw: Response 204 when post on containers

Currently POST request on container returns 202 if success. But in Swift
the code is 204. Let's use 204 to keep align with the Swift API.

Fixes #10667
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

ObjectCacher: Only r >= 0, it need the read data.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osdc: only external_call can touch object.

In fact if there is retry reading, the touch_bh also touch object.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osdc: For read w/ DONTNEED, if read data contain all cached data, move this object into the tail of LRU.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osdc: Make ObjectCahce can hanle read with DONTNEED.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osdc: touch hit bh although there are missing or rx bh.

Avoid losing the hit bh when later read, it should touch hit bh like rx bh.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osdc: Make ObjectCahce can hanle write with DONTNEED.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osdc: Make ObjectCache can accept fadvise_flags.

The field flags of OSDRead/OSDWrite don't use anymore. Replace it with
fadvise_flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osdc: add new filed dontneed in BufferHead.

For read/write op with LIBRADOS_OP_FLAG_FADVISE_DONTNEED,
the hit bh set this as true and later move into the tail of LRU in
order to remove in case in short future no ony access.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Fix ReplicatedPG do_xattr_cmp_u64

1. bufferlist.c_str() will return a string which is longer than its real length.
This could cause an error result from do_xattr_cmp_u64. So use
std::string to receive the xattr
2. s/atoll/strtoull

Signed-off-by: Dongmao Zhang <deanraccoon@gmail.com>

Merge pull request #3526 from ceph/wip-rgw-versioning-4

Wip rgw versioning 4

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: more merge related fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix merge artifact

s/XMLArgs/RGWHTTPArgs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'origin/master' into wip-rgw-versioning-4

Conflicts:
src/rgw/rgw_admin.cc

Merge pull request #3125 from ceph/wip-10194-master

Wip 10194 master

Reviewed-by: Sage Weil <sage@redhat.com>

PG: set scrubber.start = scrubber.end after scrub_compare_maps

This is how the scrubber behaved prior to
218de829b1affa8ed413572736fcc3314f65730b

Fixes: 10693
Signed-off-by: Samuel Just <sjust@redhat.com>

PG: remove block_writes from scrubber

block_writes is actually redundant, it is true iff scrubber.start !=
scrubber.end.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge branch 'prio_hb_pkts' of git://github.com/wenjianhn/ceph

Conflicts:
src/msg/Messenger.h

doc: improve incomplete state documentation

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #3543 from ceph/wip-hints

librados: rename NOREUSE to NOCACHE

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3295 from majianpeng/fix2

osd: add per-pool fadvise-dontneed flag

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3392 from ceph/wip-crush-straw2

crush: new straw2 bucket

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3495 from XinzeChi/wip-size-empty

cleanup: replace some size() or length() with !empty()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

librados: rename NOREUSE to NOCACHE

As far as I can tell, the posix_fadvise() distinction between WONTNEED and
NOREUSE is subtle: one says I won't access the data, and the other says
I will access it one more time and then not access it.  That is, the
distinction is about time.  This thread seems to confirm this
interpretation:

 https://lkml.org/lkml/2011/6/27/44

Since we are attaching hints to the IO operations themselves, this
distinction doesn't make much sense for us.  (Backends should be careful
about which hint they use; or rather, they should use WONTNEED *after*
doing the IO since NOREUSE is presenting a no-op in Linux.)

However, we want to make a totally different distinction:

 WONTNEED - nobody will access this -> drop it from the cache
 NOCACHE  - *i* won't access this again -> don't let me affect your caching
            decisions or the working set you're maintaining for other
            clients.

The NOCACHE name is made-up and distinct from NOREUSE only so that it is
different from POSIX and doesn't introduce confusion for people familiar
with the POSIX meaning.  Perhaps a more accurate name would be IWONTNEED
but that is only one character apart and too error-prone IMO.

Signed-off-by: Sage Weil <sage@redhat.com>

support Boost 1.57.0

Sometime after 1.55, boost introduced a forward declaration of
operator<< in optional.hpp. In 1.55 and earlier, when << was used
without the _io having been included, what got dumped was an implicit
bool conversion.

http://tracker.ceph.com/issues/10688 Refs: #10688
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

osd/PGLog: only do slow extra_reqid search of object appears at least once

Signed-off-by: Sage Weil <sage@redhat.com>