rgw: account common prefixes for MaxKeys in bucket listing

To be more in line with the S3 api. Beforehand we didn't account the
common prefixes towards the MaxKeys (a single common prefix counts as a
single key). Also need to adjust the marker now if it is pointing at a
common prefix.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add NextMarker param for bucket listing

Partially fixes #8858.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: improve delmited listing of bucket

If found a prefix, calculate a string greater than that so that next
request we can skip to that. This is still not the most efficient way to
do it. It'll be better to push it down to the objclass, but that'll
require a much bigger change.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

utf8: export encode_utf8() and decode_utf8()

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #2104 from ceph/wip-dencoder

move ceph-dencoder to ceph from ceph-common

Reviewed-by: Dan Mick <dan.mick@inktank.com>

Merge pull request #2114 from ceph/wip-vstart

vstart.sh: default to 3 osds

Not-NAKed-by: John Spray <john.spray@inktank.com>

test: add a missing semicolon

Broke in df8f48628.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #2119 from ceph/wip-vstart-existing-mds

Wip vstart existing mds

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2108 from kevincox/sizeint

Fix size of network protocol intergers.

Reviewed-by: Sage Weil <sage@redhat.com>

qa: generalise cephtool for vstart+MDS

Previously this test assumed no pre-existing
filesystem and no MDS running.  Generalize it
to nuke any existing filesystems found before
running, so that you can use it inside a vstart
cluster that had MDS>0.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: carry last_failure_osd_epoch across `fs new`

So that new MDSs in a new filesystem are guaranteed
to be up to date with anything we blacklisted
from a filesystem coming before.

Signed-off-by: John Spray <john.spray@redhat.com>

mon/MDSMonitor: fix msg on idempotent `fs rm`

Was outputting trailing "unrecognised command"
because we returned 0 instead of setting r=0.

Signed-off-by: John Spray <john.spray@redhat.com>

configure: do not link leveldb with everything

Detect leveldb, but do not let autoconf blindly link it with everything on the
planet.

Signed-off-by: Dan Mick <dan.mick@inktank.com>
Sighed-off-by: Sage Weil <sage@redhat.com>

AUTHORS

Signed-off-by: Sage Weil <sage@inktank.com>

doc: Add Note about European mirror in Quick Start

qa/workunits/cephtool/test.sh: fix erasure_code_profile get test

I broke this in ce9f12d7a2202948532fed9da4d763ed03f6b8fa (the pool isn't
type erasure).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2113 from ceph/wip-8857

mon/MDSMonitor: make legacy 'newfs' command idempotent

Reviewed-by: John Spray <john.spray@redhat.com>

qa/workunits/cephtool/test.sh: test osd pool get erasure_code_profile

Signed-off-by: Sage Weil <sage@inktank.com>

mon: OSDMonitor: add "osd pool get <pool> erasure_code_profile" command

Enable us to obtain the erasure-code-profile for a given erasure-pool.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>
Signed-off-by: Sage Weil <sage@inktank.com>

vstart.sh: default to 3 osds

Signed-off-by: Sage Weil <sage@inktank.com>

mon/MDSMonitor: make legacy 'newfs' command idempotent

We need to return success if we get a dup command.  Simply check whether
the fs is already enabled with the same pools and name.

Fixes: #8857
Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

rgw: don't try to wait for pending if list is empty

Fixes: #8846
Backport: firefly, dumpling

This was broken at ea68b9372319fd0bab40856db26528d36359102e. We ended
up calling wait_pending_front() when pending list was empty.
This commit also moves the need_to_wait check to a different place,
where we actually throttle (and not just drain completed IOs).

Reported-by: Sylvain Munaut <s.munaut@whatever-company.com>
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit f9f2417d7db01ecf2425039539997901615816a9)

Merge pull request #2110 from ceph/wip-8846

rgw: don't try to wait for pending if list is empty

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2095 from onlyjob/rbdmap

rbdmap: per-device post-map/pre-unmap hooks

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2107 from ceph/wip-set_layout

Wip set layout

Reviewed-by: Greg Farnum <greg@inktank.com>

rgw: don't try to wait for pending if list is empty

Fixes: #8846
Backport: firefly, dumpling

This was broken at ea68b9372319fd0bab40856db26528d36359102e. We ended
up calling wait_pending_front() when pending list was empty.
This commit also moves the need_to_wait check to a different place,
where we actually throttle (and not just drain completed IOs).

Reported-by: Sylvain Munaut <s.munaut@whatever-company.com>
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #2106 from ceph/wip-8813

eat up the level from logging output when not using the `-v` flag

Reviewed-by: Sage Weil <sage@redhat.com>

set the default log level to WARNING

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

init-ceph: wrap daemon startup with systemd-run when running under systemd

We want to make sure the daemon runs in its own systemd environment.  Check
for systemd as pid 1 and, when present, use systemd-run -r <cmd> to do
this.

Probably fixes #7627

Signed-off-by: Sage Weil <sage@redhat.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
Tested-by: Dan Mick <dan.mick@inktank.com>

doc/release-notes: v0.80.4

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2101 from ceph/wip-mds-session-asok-squash

Provide client session handling and information via the admin socket.

Reviewed-by: Greg Farnum <greg@inktank.com>

Fix size of network protocol intergers.

This fixes the size of some integers that are visible in the network
protocol.  There should be no change for machines where sizeof(int) ==
4.

Signed-Of-By: Kevin Cox <kevincox@kevincox.ca>

doc: add cephfs layout documentation

This clarifies how to deal with layouts in CephFS
using vxattrs.  We can point people here if they
ask what they should use instead of the deprecated
`cephfs set_layout`.

Signed-off-by: John Spray <john.spray@redhat.com>

cephfs: pool safety & print deprecation warning

Print a message indicating to users that this tool
is now deprecated in favour of vxattrs.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: retire kclient-specific layout test

`cephfs set_layout` was broken and is now deprecated
in favour of using xattrs for layout.  Retire the
kclient-specific test.

Fixes: #8773
Signed-off-by: John Spray <john.spray@redhat.com>

ceph.spec: move ceph-dencoder to ceph from ceph-common

Signed-off-by: Sage Weil <sage@redhat.com>

debian: move ceph-dencoder to ceph from ceph-common

It links against the world currently (notably leveldb).  Not nice for the
client-side lib.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2098 from ceph/wip-dump-new-crush

crush: include CRUSH_V3, v2/v3 rules checks in dump_tunables()

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #2103 from ceph/wip-8830

XfsFileStoreBackend: default to disabling extsize on xfs

Reviewed-by: Sage Weil <sage@redhat.com>

XfsFileStoreBackend: default to disabling extsize on xfs

This appears to be responsible for the deep scrub mismatches on some rbd
workloads.

Fixes: 8830
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2102 from ceph/wip-8823

ceph_test_rados_api_tier: fix [EC] HitSet{Read,Write,Trim} tests

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #2097 from sjahl/master

doc: Fixes a broken link on the rados deploy osd page.

Reviewed-by: John Wilkins <john.wilkins@inktank.com>

doc/release-notes: some additional warnings and recommendations against adjusting tunables

Several users are having troubles related to the overhead of the data
rebalance from adjusting tunables.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_tier: fix [EC] HitSet{Read,Write,Trim} tests

The hit_set_ fields can only be set on tier pools as of
f131dfbaedf6f451572e7aa3a83f653912122953.

Fixes: #8823
Signed-off-by: Sage Weil <sage@inktank.com>

mds: add `session ls` and `session evict` to asok

These commands are intended to help admins deal
with MDSs during recovery, to identify troublesome
clients which may need intervention (such as eviction).

Signed-off-by: John Spray <john.spray@redhat.com>

client: include ID in mds_sessions asok

So that an admin can map a ceph-fuse process
reliably to an entry in the MDS session list.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: remove unused attribute notified_global_id

Signed-off-by: John Spray <john.spray@redhat.com>

common/admin_socket: remove dead code

Signed-off-by: John Spray <john.spray@redhat.com>

osd: fix confusing debug output for op_applied

Signed-off-by: Sage Weil <sage@inktank.com>

doc: Fixes a broken link on the rados deploy osd page.

Signed-off-by: Stephen Jahl <stephenjahl@gmail.com>

Merge pull request #1999 from yuyuyu101/keyvaluestore-cache

Keyvaluestore cache

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2096 from ceph/wip-test-post-file

qa/workunits/post-file.sh

Reviewed-by: Greg Farnum <greg@inktank.com>

doc/release-notes: v0.80.3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/post-file.sh

Test ceph-post-file.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2090 from ceph/wip-nuke-dlist

nuke dlist

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #2089 from lpabon/lpabon_readforward

Support for READFORWARD in the caching tier

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2093 from theanalyst/doc/format-fix

doc: minor format fix for radosgw admin docs

Merge pull request #2042 from ceph/wip-port-fixes

Portability updates

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2086 from ceph/wip-8696

mon: OSDMonitor: limit tier-specific pool set/get on non-tier pools

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2091 from ceph/wip-kinetic-os

prototype kinetic osd backend

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #2071 from somnathr/wip-sd-cpu-optimized

Wip sd cpu optimized

Reviewed-by: Samuel Just <sam.just@inktank.com>

rbdmap: per-device post-map/pre-unmap hooks

There are cases when automatic (un-)mounting of file system on RBD is not
enough. Some services may need to be started when RBD device becomes
available (mapped) as well as it may be desirable to stop services in order
to release file system before unmapping RBD device.

File system(s) on RBD is not the only use case scenario. RBD devices may be
used as block devices in which case `/etc/fstab` is not sufficient to
perform action upon mapping RBD device. A handler script (hook) can be
useful to properly release RBD device before unmapping, etc.

Pre-unmap hooks can be important for clean shut down and for re-exporting
RBD device(s) as (iSCSI,AoE,DRBD) etc.

This commit introduces support for per-device hooks to perform per-device
post-map/pre-unmap actions. If hook named like "poolname/imagename" (same
as in `/etc/ceph/rbdmap` file) is found in

    /etc/ceph/rbd.d/poolname/imagename

it is executed as

    /etc/ceph/rbd.d/poolname/imagename map poolname/imagename

following after attempt to mount file system (if relevant `fstab` entry
exist) following mapping of corresponding RBD device.

Before un-mounting file system and un-mapping RBD device hook is called as
follows:

    /etc/ceph/rbd.d/poolname/imagename unmap poolname/imagename

Second argument is intentional to allow multiple RBD devices to share the
same hook (symlinked under different names).

Sample hook to use RBD device as "mdadm" hot spare may look like this:

~~~~
DEV="$2"
case "$1" in
    "map")
        mdadm /dev/md2 --add --write-mostly "${DEV}"
    ;;
    "unmap")
        mdadm /dev/md2 --fail "${DEV}"
        sleep 2
        mdadm /dev/md2 --remove "${DEV}"
    ;;
esac
~~~~

Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>

Merge pull request #2094 from ceph/wip-8815

mon: avoid scrub noise due to divering OSDMap encoding versions

Reviewed-by: Greg Farnum <greg@inktank.com>

osd/osd_types: be pedantic about encoding last_force_op_resend without feature bit

The addition of the value is completely backward compatible, but if the
mon feature bits don't match it can cause monitor scrub noice (due to the
parallel OSDMap encoding).  Avoid that by only adding the new field if the
feature (which was added 2 patches after the encoding, see
3152faf79f498a723ae0fe44301ccb21b15a96ab and
45e79a17a932192995f8328ae9f6e8a2a6348d10.

Fixes: #8815
Backport: firefly
Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: remove useless encode_compat local var for pg_pool_t::encode

This is left over from db3fd1152a87f8f152facdf141f8367296b371df
when we made the HitSet stuff incompat, but that behavior was reverted by
dc3ce58add130ca5a0a20b097b0ac934a1b486c7, leaving this cruft behind.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2088 from guangyy/wip-8733

Fix the PG listing issue which could miss objects for EC pool

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2092 from onlyjob/rbdmap

rbdmap: per-device mount (Closes: #8538)

Reviewed-by: Sage Weil <sage@redhat.com>

qa/workunits: cephtool: adjust pool name where missing as it has changed

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

crush: include CRUSH_V3, v2/v3 rules checks in dump_tunables()

Backport: firefly
Signed-off-by: Sage Weil <sage@inktank.com>

doc: minor format fix for radosgw admin docs

Missing backtick fixed

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

rbdmap: per-device mount (Closes: #8538)

`/etc/init.d/rbdmap start` was doing `mount -a`. Although (arguably)
`mount -a -O _netdev` could be less disruptive, it's not RBD mapping job to
mount unrelated devices and potentially do it at the wrong time.

Solution is to call `mount {device}` which works as expected and mounts
device even if it given in form `mount /dev/rbd/pool/imagename` while
`/etc/fstab` uses UUID or LABEL notation.

Furthermore this commit

 * fixes global exit code (it was always 0): now it is 0 only when
   all devices were (un)mounted successfully; otherwise non-zero.
 * replaces `mount -a` with per-device post-mapping `mount {dev}`
 * show mapping progress using LSB functions per device instead of for
   {start|stop} invocation.
 * capture output of `(u)mount` (if any) and report it as "info".

Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>

rbd.cc: Check io-size avoid floating point exception.

In func do_bench_write if io_size is zero,it can cause floating point execption.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

qa/workunits: cephtool: cleanup after pool creation

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunits: cephtool: pool needs to be a tier to be used as such

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunits: cephtool: test erroneous 'tier remove'

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunits: cephtool: test get/set on both tier and non-tier pools

Make sure gets and sets of tiering-specific variables succeed on tier
pools and fail on non-tier pools.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunits: cephtool: split get/set on tier pools from get/set tests

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: OSDMonitor: be scary about inconsistent pool tier ids

We may not crash your cluster, but you'll know that this is not something
that should have happened.  Big letters makes it obvious.  We'd make them
red too if we bothered to look for the ANSI code.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

osd: pg_pool_t: clear tunables on clear_tier()

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: OSDMonitor: limit tier-specific pool set/get on non-tier pools

Fixes: 8696
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

doc/changelog/v0.80.2: include tag

Signed-off-by: Sage Weil <sage@inktank.com>

os: add prototype KineticStore

Implement the KeyValueDB interface using libkinetic_client,
and allow it to be configured as the backend for the KeyValueStore,
running the entire OSD on it.

This prototype implementation has no transaction safety, and is
only suitable as a proof of concept. Since the libkinetic_client
API does not provide reverse iteration over keys without also reading
the value off disk, it implements iterators in a very slow but correct way.
These are used heavily by the KeyValueDB callers, so this is a bottleneck
in performance.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

PG::op_must_wait_for_map: pass the epoch rather than the map

have_same_or_newer_map is also modified to accept epoch instead of
map.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

Merge branch 'master' of https://github.com/ceph/ceph

doc: Added CentOS/RHEL install for git.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

rgw: modelines

Signed-off-by: Sage Weil <sage@inktank.com>

rgw: fix RGWObjManifestRule decoder

Only decode the new field if it is a new struct.

Fixes: #8804
Backport: firefly
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

doc: Added keyring location note to resolve pull abandoned pull request #1946.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

nuke dlist

No users.

Signed-off-by: Sage Weil <sage@inktank.com>

Add random_cache.hpp to Makefile.am

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

test: fix make_pair() for c++11

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

Remove some not-resolving make_pair() invocations.

There's nothing wrong with make_pair(), but there is some type
resolution issue in these instances, at least with GCC 4.8.

Signed-off-by: Matt Benjamin <matt@linuxbox.com>

Add required spacing after string literals (PRIu64).

The common formatting for these 64bit types is not C++ 2011
clean.

Signed-off-by: Matt Benjamin <matt@linuxbox.com>
Conflicts:
src/common/perf_counters.cc

Convert ConnectionRef explicitly to bool.

Signed-off-by: Matt Benjamin <matt@linuxbox.com>

Work around an apparent binding bug (GCC 4.8).

A reference to h->seq passed to std::pair ostensibly could not bind
because the header structure is packed.  At first this looked like
a more general unaligned access problem, but the only location the
compiler rejects is a false positive.

Signed-off-by: Matt Benjamin <matt@linuxbox.com>

Fix the PG listing issue which could miss objects for EC pool (where there is object shard and generation).
Backport: firefly
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

doc: Clean up formatting, usage and removed duplicate section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc/release-notes: v0.80.2

Signed-off-by: Sage Weil <sage@redhat.com>