rgw_admin: add a few admin commands to the usage

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: add bi purge command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket resharding, adjust logging

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: bi_list() fix is_truncated returned param

is_truncated was never set. Also, make sure that we don't return
more entries than requested.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: require --yes-i-really-mean-it for bucket reshard

in the case where num shards are less or equal to current bucket

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: better bucket reshard logging

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: limit bucket reshard num shards to max possible

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: fix bi list command

Changes scoping of json section, and push the --object param in so that
an object can be specified as a filter.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: use aio operations for bucket resharding

also created shards manager to make things slightly cleaner

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket reshard updates stats

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: add bucket_update_stats method

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: reshard also links to new bucket instance

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rgw_link_bucket, use correct bucket structure for entry point

The bucket structure might be different than the one that we were using
before.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: bucket reshard needs --num-shards to be specified

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: fix bi_list objclass command

was filtering entries, even if filter was not specified, and need to
set boundary for plain entries. Also, list_instance_entries() was not
working correctly, and added list_olh_entries().

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: bucket rehsrading, initial work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: utilities to support raw bucket index operations

and other related changes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use bucket_info.bucket_id instead of marker where needed

We used to use these interchangeably, but they actually have diffent meaning.
The marker is the prefix we assign to the objects in this bucket, whereas
the bucket_id is the bucket instance's id. These used to hold the same
value, but with bucket resharding it's not going to be true anymore.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: utilities to support raw bucket index operations

New flavour of bi_put() call, and a function to extract key off
a raw bi entry.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11546 from umullan/uday

doc: Fix broken link for caps

Merge pull request #11558 from ifed01/wip-rmcoll-more-fixes

os/memstore, os/filestore: fix store_test's to satisfy rm_coll behavi…

Reviewed-by: Sage Weil <sage@redhat.com>

os/memstore, os/filestore: fix store_test's to satisfy rm_coll behavior expected in store_test

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11207 from yehudasa/wip-17372

rgw: get_zonegroup() uses "default" zonegroup if empty
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10919 from pritha-srivastava/wip-17076

rgw: Replacing '+' with "%20" in canonical uri for s3 v4 auth.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11316 from stiopaa1/mds_sessionMapUnnamedNamespace

mds/SessionMap.cc:put classes in unnamed namespace

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11450 from stiopaa1/mds_damagetable_moveClassesToCCfile

mds/DamageTable: move classes to .cc file

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11511 from stiopaa1/mds_mdsmap_addConstToMemFunctions

mds/MDSMap.h: add const to member functions

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11550 from renhwztetecs/renhw-wip-mgr-doc

doc/mgr: fix mgr how long to wait to failover

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11304 from ajarr/wip-volume-client-recovery-fixes

ceph_volume_client: fix recovery from partial auth update

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11386 from xiexingguo/xxg-wip-mgr-1008

mgr: misc minor changes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11507 from ceph/wip-17579

build/ops: deb: add python dependencies where needed

Reviewed-by: Alfredo Deza <adeza@redhat.com>

build/ops: deb: add python dependencies where needed

this reverts 48f4576 and f140a8c

Fixes: http://tracker.ceph.com/issues/17579
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

build/ops: deb: put depends on separate lines

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11557 from xiexingguo/xxg-wip-bluestore-1019

os/bluestore: kill overlay related options

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11516 from david-z/wip-add-force-ro-ceph-string

common/ceph_string: add ceph string constants for CEPH_SESSION_FORCE_RO

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11552 from tchaikov/wip-fix-enc

include/object: pass "snapid_t&" to bound_encode()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11434 from batrick/mds-cmd-deprecated

Flag deprecated mds commands and omit deprecated mon commands in help output

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11486 from batrick/client-gid

client: get gid from MonClient

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11373 from ukernel/wip-mds-recall-warning

mds: fix false "failing to respond to cache pressure" warning

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11551 from wjwithagen/wip-wjw-freebsd-lltng

do_freebsd.sh: Do not use LTTNG on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

include/object: pass "snapid_t&" to bound_encode()

otherwise the STL container's encoders will segfault when trying to
figure out the element size: they use "denc(*(A*)nullptr, elem_size)"
to do the calculation.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/mgr: fix mgr how long to wait to failover

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

do_freebsd.sh: Do not use LLTNG on FreeBSD

  FreeBSD has Dtrace

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #11534 from ceph/wip-17607-ceph-disk

tests: ceph-disk: force debug monc = 0

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: kill overlay related options

These two options are not used anymore.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Fix broken link for caps
http://tracker.ceph.com/issues/17587
Signed-off-by: Uday udaymjl@gmail.com

Merge pull request #11512 from wjwithagen/wip-wjw-freebsd-depends

FreeBSD: update require packages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11204 from liewegas/wip-rados-ack

librados: do not request osd ack if no completed completion is set

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #11213 from liewegas/wip-osd-fastinfo

osd: add a pg _fastinfo attribute to reduce per-io metadata updates

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #11354 from liewegas/wip-rocksdb

rocksdb: update to latest, and make it the default for the mons

Merge pull request #11500 from liewegas/wip-lttng

CMakeLists.txt: enable LTTNG by default

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11540 from liewegas/wip-bluestore-cow

os/bluestore: default bluestore_clone_cow=true

Merge pull request #11539 from liewegas/wip-bluestore-gc

os/bluestore: no garbage collection for uncompressed blobs

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: default bluestore_clone_cow=true

This has been stable for a long time.

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "ceph_test_objecstore: use 1TB block device"

This reverts commit 5389133ff9a7783d77796ed73893e26f34c6e5d2.

This is too slow with the bitmap allocator.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: No garbage collection for uncompressed blobs

Signed-off-by: Roushan Ali<roushan.ali@sandisk.com>

Merge pull request #11536 from liewegas/wip-fullbit-dump

mds: fix EMetaBlob::fullbit xattr dump

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11533 from ifed01/wip-kstore-rmcoll-fix

os/kstore: rmcoll fix to satisfy store_test

Reviewed-by: Sage Weil <sage@redhat.com>

mds: fix EMetaBlob::fullbit xattr dump

- show xattr names!
- bound the values

Signed-off-by: Sage Weil <sage@redhat.com>

tests: ceph-disk: force debug monc = 0

The sh function will collect both stderr and stdout and debug
will mess the json parsing.

Fixes: http://tracker.ceph.com/issues/17607
Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11525 from wjwithagen/wip-wjw-freebsd-getopt

tools/ceph-monstore-update-crush.sh: FreeBSD getopt is not compatible…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11524 from wjin/filestore_perfcounter

os/filestore: clean filestore perfcounters

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11518 from umullan/uday

doc: fix broken links

Merge pull request #11522 from wjwithagen/wip-wjw-freebsd-readable.sh

FreeBSD:test/encoding/readable.sh": fix nproc and ls -v calls

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11528 from jcsp/wip-vstart-1osd

vstart: decrease pool size if <3 OSDs

Reviewed-by: Sage Weil <sage@redhat.com>

osd/PG: add perfcounters for pg info updates

Note info updates (of any kind), fastinfo updates,
and biginfo updates.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: proper set resulting pnext to avoid returning local var

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/kstore: fix KStore rm_coll handler to behave similar to bluestore one

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

ceph-objectstore-tool: relax cur_struct_v checks

All compat pg formats can handle past interval removal
and mark-complete.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: make fastinfo apply conditionally

This way we don't have to bother deleting the fastinfo
attr, which will reduce the amount of work the
underlying store has to do.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add _fastinfo PG attr for common pg_info_t updates

For most IO operations we only update a handful of fields
in the pg_info_t structure.  However, the full struct,
when encoded, is on the order of 800 bytes.

This adds a new attribute, _fastinfo, which contains only
the most commonly updated fields.  When present, the
fastinfo fields should be overlayed on top of the full
info struct contained in the existing info attr.  If
a field outside of the "fast" set is updated, we clear
the fastinfo attribute and update the full info attr.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: pass last_writen_info to _prepare_write_info

We'll use this shortly.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: operator== for pg hit sets, history, info

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop unused args to _prepare_write_info

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: pg_stat_t ondisk_log_size == log_size

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10325 from snakeAngel2015/rgw-support-s3subuser

rgw: add suport for creating S3 type subuser of admin rest api
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

os/kstore: uniform Collection implementation with BlueStore one

Signed-off-by:Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11481 from ddiss/cmake_rgw_deps

cmake: check WITH_RADOSGW for fcgi and expat dependencies

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart: decrease pool size if <3 OSDs

Makes it easy to do OSD=1 for e.g. doing cephfs
stuff where we don't care much about the OSDs
behaviour.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #10239 from weiqiaomiao/wqm-wip-setolh

rgw: should assign 'olh_bl" to state.attrset[RGW_ATTR_OLH_ID_TAG] instead of 'bl'
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11324 from yehudasa/wip-14589

rgw: dump objects in RGWBucket::check_object_index()
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

tools/ceph-monstore-update-crush.sh: FreeBSD getopt is not compatible, use the one from packages

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #11432 from hansbogert/patch-3

doc: Update install-ceph-gateway.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

Fixes: #17587
doc: Fixing the broken link which was refactored for authentication as part of other PR
Signed-off-by: uday udaymjl@gmail.com

os/filestore: clean filestore perfcounters

Move filestore perfcounters from objectstore to filestore itself
and change internal index name.

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

Merge pull request #11027 from liewegas/wip-denc

fast denc encoding

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

Merge remote-tracking branch 'gh/kraken'

cmake: find GIT_VER variables if there is no .git dir

find CEPH_GIT_VER & CEPH_GIT_NICE_VER if there is no
ceph/.git dir by parsing ceph/src/.git_version.

scenario is for building from source tarballs generated
by make-dist, since make-dist generates .git_version
by running make_version, and then packs .git_version
into the tarball.

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit c5c6b59f296f0ace3a6cfd939027118e82ab50a4)

Merge pull request #11502 from liewegas/wip-bluestore-keybug

os/bluestore: fix escaping of chars > 0x80

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/kstore: fix escaping of chars > 0x80 in keys

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix escaping of odd chars >0x80 in keys

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: test unprintable chars

Signed-off-by: Sage Weil <sage@redhat.com>

FreeBSD:test/encoding/readable.sh": fix nproc and ls -v calls

 Neither of these is available in FreeBSD.
 But sort -n is Posix, and does the same ATM.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #11521 from markhpc/wip-send_inline

msg/async: set ms_async_send_inline to false to improve small randread iops

Set ms_async_send_inline to false to improve small randread iops.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

mgr: shutdown finisher in a graceful way

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/MgrMonitor: make period of sending digests configurable

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/MgrMonitor: break loop on catching specified standby

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/MgrMonitor: unify updating map code path

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/MgrStandby: do not ignore result of init() of monc

Which can fail due to missing keyring.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>