msg/Message: fix set_middle vs throttler

Signed-off-by: Sage Weil <sage@redhat.com>

msg: adjust byte_throttler from Message::encode

Normally we never call encode on a message that has a byte_throttler set
because we only use it for messages we received.  However, for forwarded
messages that we clear_payload() before resending, we *do* reencode, and in
that case we need to retake the appropriate number of bytes from the
throttler--just like we release them in clear_payload().

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MForward: reencode forwarded message if target has differing features

This ensures we reencode the payload with the
appropriate set of features if the client, us, or the
target do not have identical features.  Otherwise we
may forward an encoding with more features than the
target can handle.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11607 from xiexingguo/xxg-wip-bluestore-1022

os/bluestore: simplify can_split_at()

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: literal changes
1. drop redundant semicolon
2. fix wrong output message

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: simplify can_split_at()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11026 from jcsp/wip-pg-files

tools/cephfs: add pg_files command

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11456 from tchaikov/wip-17558

mon: send updated monmap to its subscribers

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11599 from tanghaodong25/wip-2q-cache

os/bluestore: make 2q cache kin/kout size tunable

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11603 from liewegas/wip-bluestore-obj-remove

os/bluestore: clear extent map on object removal

Merge pull request #11597 from ifed01/wip-bluestore-reshard-fix

os/bluestore: fix lack of resharding

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: clear extent map on object removal

Clear ExtentMap (esp shards, etc.) when an object is removed.  Otherwise
if we recreate it we will have stale state (like the shards vector or
inline_bl) that are bogus.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11230 from yehudasa/wip-rgw-resharding

rgw: bucket resharding
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

os/bluestore: make 2q cache kin/kout size tunable

default kin/hot queue size is 50% of max buffer size.
50% is a better choice for the number of page slot in kout.
we make it tunable.

Signed-off-by: Haodong Tang <haodong.tang@intel.com>

os/bluestore: fix black of resharding

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11566 from oritwas/wip-rgw-put-acl-underscore

rgw: fix put_acls for objects starting and ending with underscore

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11591 from liewegas/wip-bluestore-test-dev

ceph_test_objectstore: smaller device

Merge pull request #11444 from cbodley/wip-rgw-get-system-obj

rgw: get_system_obj does not use result of get_system_obj_state
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11588 from xiexingguo/wip-fix-fastinfo-id

osd: fix duplicated id of incompat feature "fastinfo"

Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: smaller device

This is faster.  The bitmap has to be scanned during
startup and fsck and that's slow.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11575 from ceph/wip-17260

cmake: link tests against static librados

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10490 from Yan-waller/yj-wip-msgtest-0729

msg/test: fix the guided compile-command to ceph_test_msgr

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: fix duplicated id of incompat feature "fastinfo"

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11436 from chhabaramesh/master

kv/MemDB: making memdb code adapt to generic maps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11455 from stiopaa1/mds_mutation_simplifyConstructors

mds/Mutation.h: simplify constructors

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11509 from stiopaa1/mds_fsmapuser_makeAssignCopyDefault

mds/FSMapUser.h: remove copy ctr and assign op

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10359 from songbaisen/b11

mon:  paxos add the timeout function when peon recovery

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #11165 from xiexingguo/xxg-wip-throttle-0921

common/throttle: simplify Throttle::_wait()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11583 from stiopaa1/mds_snapserver_addOverride

mds/SnapServer.h: add override to virtual functs

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11523 from stiopaa1/mds_migrator_removeUnneededCount

mds/Migrator.cc: remove unneeded use of count

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11543 from batrick/cdir-unique-ptr

mds: use unique_ptr to simplify resource mgmt

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11581 from yehudasa/wip-17635

rgw: handle empty POST condition

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11565 from stiopaa1/mds_sessionmap_removeUnneededFunction

mds/SessionMap.h: remove unneeded function

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rgw: handle empty POST condition

Fixes: http://tracker.ceph.com/issues/17635
Before accessing json entity, need to check that iterator is valid.
If there is no entry return appropriate error code.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mds/SnapServer.h: add override to virtual functs

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11580 from cbodley/wip-rgw-deadlock-dump-crs

rgw: dump remaining coroutines when cr deadlock is detected

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11526 from ukernel/wip-17591

client: get caller's uid/gid on every libcephfs operation

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rgw: dump remaining coroutines when cr deadlock is detected

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11553 from Aran85/wip-datasync-status

rgw multisite: fix the increamtal bucket sync init

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11577 from jcsp/wip-17636

denc: don't pass null instances into encoder fns

Reviewed-by: Kefu Chai <kchai@redhat.com>

include/fs_types: fix crash in inodeno_t encode

Passing by value tries to copy from a null and segfaults.

Fixes: http://tracker.ceph.com/issues/17636
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #11055 from gcharot/ceph-lazy

Adding ceph-lazy tool

Reviewed-by: Sébastien Han <han.sebastien@gmail.com>

Merge pull request #8316 from dx9/wip-ceph-detect-init

ceph_detect_init: add support for Alpine

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11520 from dillaman/wip-17590

journal: do not prematurely flag object recorder as closed

Merge pull request #11480 from dillaman/wip-17566

test: TestJournalReplay test cases need to wait for event commit

Reviewed-by: Venky Shankar <vshankar@redhat.com>

test/encoding: do not include unused headers

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link against rados_a instead of its objs

prior to this change, librbd tests link against librados_api_obj and
librados_objs, but since librados_a offers all the goodies, we can
just link against it instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: librados depends on osdc implicitly

so no need to put add_dependencies() explicitly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not link against global if not necessary

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link tests against static librados

Fixes: http://tracker.ceph.com/issues/17260
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: always create static lib librados_a

so tests referencing internal symbols can use librados_a instead of
librados to avoid violating ODR (one definition rule).

Fixes: http://tracker.ceph.com/issues/17260
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11175 from asheplyakov/wip-rbd-default-features

librbd: parse rbd_default_features config option as a string

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw multisite: fix the increamtal bucket sync init

in the `RGWBucketShardFullSyncCR::operate`, inc_marker will assigned with remote bilog's max_marker.
but the sync_status's inc_marker cant be assigned.so the next step inc sync will always sync
from null log,which means at beginning log.

Fixes: http://tracker.ceph.com/issues/17624
Signed-off-by: Zengran Zhang <zhangzengran@h3c.com>

Merge pull request #11567 from ceph/wip-master-rgw-0lenfix2

Don't loop forever when reading data from 0 sized segment.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Don't loop forever when reading data from 0 sized segment.

The 0 sized segment can arise depending on how
the client uploads the object in the first place.
The cpu loop then happens during a swift `GET'.

Signed-off-by: Marcus Watts <mwatts@redhat.com>

tools/cephfs: add pg_files command

Sometimes users know that particular data pool PGs
have been damaged, and they would like to scan
their files to work out which ones might have
been affected.

Fixes: http://tracker.ceph.com/issues/17249
Signed-off-by: John Spray <john.spray@redhat.com>

libcephfs: take ref when constructed with CephContext

Previously, if you used ceph_create_with_context, then
it would try and destroy your context when you called
ceph_release.  Bad behaviour: caller created the
context, caller should decide when to destroy it.

Signed-off-by: John Spray <john.spray@redhat.com>

rgw: fix put_acls for objects starting and ending with underscore

Fixes: http://tracker.ceph.com/issues/17625
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11541 from stiopaa1/mds_sessionmap_addConst

mds/SessionMap: add const to member functions

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

unittest_bluestore_types: add random put_ref test case

This appeared to fail under load, but works fine here.
:/

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: fix gc tests

In ef8989ce7417034aa715de4fe83d1f0363c044f4 we disabled
gc for non-compressed blobs.  So mark these test ones as
compressed.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw_admin: add a few admin commands to the usage

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: add bi purge command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket resharding, adjust logging

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: bi_list() fix is_truncated returned param

is_truncated was never set. Also, make sure that we don't return
more entries than requested.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: require --yes-i-really-mean-it for bucket reshard

in the case where num shards are less or equal to current bucket

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: better bucket reshard logging

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: limit bucket reshard num shards to max possible

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: fix bi list command

Changes scoping of json section, and push the --object param in so that
an object can be specified as a filter.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: use aio operations for bucket resharding

also created shards manager to make things slightly cleaner

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket reshard updates stats

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: add bucket_update_stats method

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: reshard also links to new bucket instance

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rgw_link_bucket, use correct bucket structure for entry point

The bucket structure might be different than the one that we were using
before.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: bucket reshard needs --num-shards to be specified

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: fix bi_list objclass command

was filtering entries, even if filter was not specified, and need to
set boundary for plain entries. Also, list_instance_entries() was not
working correctly, and added list_olh_entries().

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: bucket rehsrading, initial work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11438 from trociny/wip-17488

rbd-nbd: support disabling auto-exclusive lock transition logic

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: utilities to support raw bucket index operations

and other related changes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use bucket_info.bucket_id instead of marker where needed

We used to use these interchangeably, but they actually have diffent meaning.
The marker is the prefix we assign to the objects in this bucket, whereas
the bucket_id is the bucket instance's id. These used to hold the same
value, but with bucket resharding it's not going to be true anymore.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: utilities to support raw bucket index operations

New flavour of bi_put() call, and a function to extract key off
a raw bi entry.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mds: use unique_ptr to simplify resource mgmt

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11546 from umullan/uday

doc: Fix broken link for caps

Merge pull request #11558 from ifed01/wip-rmcoll-more-fixes

os/memstore, os/filestore: fix store_test's to satisfy rm_coll behavi…

Reviewed-by: Sage Weil <sage@redhat.com>

os/memstore, os/filestore: fix store_test's to satisfy rm_coll behavior expected in store_test

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11207 from yehudasa/wip-17372

rgw: get_zonegroup() uses "default" zonegroup if empty
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10919 from pritha-srivastava/wip-17076

rgw: Replacing '+' with "%20" in canonical uri for s3 v4 auth.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11316 from stiopaa1/mds_sessionMapUnnamedNamespace

mds/SessionMap.cc:put classes in unnamed namespace

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11450 from stiopaa1/mds_damagetable_moveClassesToCCfile

mds/DamageTable: move classes to .cc file

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11511 from stiopaa1/mds_mdsmap_addConstToMemFunctions

mds/MDSMap.h: add const to member functions

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11550 from renhwztetecs/renhw-wip-mgr-doc

doc/mgr: fix mgr how long to wait to failover

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11304 from ajarr/wip-volume-client-recovery-fixes

ceph_volume_client: fix recovery from partial auth update

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11386 from xiexingguo/xxg-wip-mgr-1008

mgr: misc minor changes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11507 from ceph/wip-17579

build/ops: deb: add python dependencies where needed

Reviewed-by: Alfredo Deza <adeza@redhat.com>

build/ops: deb: add python dependencies where needed

this reverts 48f4576 and f140a8c

Fixes: http://tracker.ceph.com/issues/17579
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

build/ops: deb: put depends on separate lines

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11557 from xiexingguo/xxg-wip-bluestore-1019

os/bluestore: kill overlay related options

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11516 from david-z/wip-add-force-ro-ceph-string

common/ceph_string: add ceph string constants for CEPH_SESSION_FORCE_RO

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11552 from tchaikov/wip-fix-enc

include/object: pass "snapid_t&" to bound_encode()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11434 from batrick/mds-cmd-deprecated

Flag deprecated mds commands and omit deprecated mon commands in help output

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>