os/bluestore: make 2q cache kin/kout size tunable

default kin/hot queue size is 50% of max buffer size.
50% is a better choice for the number of page slot in kout.
we make it tunable.

Signed-off-by: Haodong Tang <haodong.tang@intel.com>

Merge pull request #11575 from ceph/wip-17260

cmake: link tests against static librados

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10490 from Yan-waller/yj-wip-msgtest-0729

msg/test: fix the guided compile-command to ceph_test_msgr

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11436 from chhabaramesh/master

kv/MemDB: making memdb code adapt to generic maps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11455 from stiopaa1/mds_mutation_simplifyConstructors

mds/Mutation.h: simplify constructors

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11509 from stiopaa1/mds_fsmapuser_makeAssignCopyDefault

mds/FSMapUser.h: remove copy ctr and assign op

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10359 from songbaisen/b11

mon:  paxos add the timeout function when peon recovery

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #11165 from xiexingguo/xxg-wip-throttle-0921

common/throttle: simplify Throttle::_wait()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11583 from stiopaa1/mds_snapserver_addOverride

mds/SnapServer.h: add override to virtual functs

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11523 from stiopaa1/mds_migrator_removeUnneededCount

mds/Migrator.cc: remove unneeded use of count

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11543 from batrick/cdir-unique-ptr

mds: use unique_ptr to simplify resource mgmt

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11581 from yehudasa/wip-17635

rgw: handle empty POST condition

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11565 from stiopaa1/mds_sessionmap_removeUnneededFunction

mds/SessionMap.h: remove unneeded function

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rgw: handle empty POST condition

Fixes: http://tracker.ceph.com/issues/17635
Before accessing json entity, need to check that iterator is valid.
If there is no entry return appropriate error code.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mds/SnapServer.h: add override to virtual functs

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11580 from cbodley/wip-rgw-deadlock-dump-crs

rgw: dump remaining coroutines when cr deadlock is detected

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11526 from ukernel/wip-17591

client: get caller's uid/gid on every libcephfs operation

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rgw: dump remaining coroutines when cr deadlock is detected

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11553 from Aran85/wip-datasync-status

rgw multisite: fix the increamtal bucket sync init

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11577 from jcsp/wip-17636

denc: don't pass null instances into encoder fns

Reviewed-by: Kefu Chai <kchai@redhat.com>

include/fs_types: fix crash in inodeno_t encode

Passing by value tries to copy from a null and segfaults.

Fixes: http://tracker.ceph.com/issues/17636
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #11055 from gcharot/ceph-lazy

Adding ceph-lazy tool

Reviewed-by: Sébastien Han <han.sebastien@gmail.com>

Merge pull request #8316 from dx9/wip-ceph-detect-init

ceph_detect_init: add support for Alpine

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11520 from dillaman/wip-17590

journal: do not prematurely flag object recorder as closed

Merge pull request #11480 from dillaman/wip-17566

test: TestJournalReplay test cases need to wait for event commit

Reviewed-by: Venky Shankar <vshankar@redhat.com>

test/encoding: do not include unused headers

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link against rados_a instead of its objs

prior to this change, librbd tests link against librados_api_obj and
librados_objs, but since librados_a offers all the goodies, we can
just link against it instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: librados depends on osdc implicitly

so no need to put add_dependencies() explicitly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not link against global if not necessary

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link tests against static librados

Fixes: http://tracker.ceph.com/issues/17260
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: always create static lib librados_a

so tests referencing internal symbols can use librados_a instead of
librados to avoid violating ODR (one definition rule).

Fixes: http://tracker.ceph.com/issues/17260
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11175 from asheplyakov/wip-rbd-default-features

librbd: parse rbd_default_features config option as a string

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw multisite: fix the increamtal bucket sync init

in the `RGWBucketShardFullSyncCR::operate`, inc_marker will assigned with remote bilog's max_marker.
but the sync_status's inc_marker cant be assigned.so the next step inc sync will always sync
from null log,which means at beginning log.

Fixes: http://tracker.ceph.com/issues/17624
Signed-off-by: Zengran Zhang <zhangzengran@h3c.com>

Merge pull request #11567 from ceph/wip-master-rgw-0lenfix2

Don't loop forever when reading data from 0 sized segment.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Don't loop forever when reading data from 0 sized segment.

The 0 sized segment can arise depending on how
the client uploads the object in the first place.
The cpu loop then happens during a swift `GET'.

Signed-off-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #11541 from stiopaa1/mds_sessionmap_addConst

mds/SessionMap: add const to member functions

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

unittest_bluestore_types: add random put_ref test case

This appeared to fail under load, but works fine here.
:/

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: fix gc tests

In ef8989ce7417034aa715de4fe83d1f0363c044f4 we disabled
gc for non-compressed blobs.  So mark these test ones as
compressed.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11438 from trociny/wip-17488

rbd-nbd: support disabling auto-exclusive lock transition logic

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mds: use unique_ptr to simplify resource mgmt

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11546 from umullan/uday

doc: Fix broken link for caps

Merge pull request #11558 from ifed01/wip-rmcoll-more-fixes

os/memstore, os/filestore: fix store_test's to satisfy rm_coll behavi…

Reviewed-by: Sage Weil <sage@redhat.com>

os/memstore, os/filestore: fix store_test's to satisfy rm_coll behavior expected in store_test

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11207 from yehudasa/wip-17372

rgw: get_zonegroup() uses "default" zonegroup if empty
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10919 from pritha-srivastava/wip-17076

rgw: Replacing '+' with "%20" in canonical uri for s3 v4 auth.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11316 from stiopaa1/mds_sessionMapUnnamedNamespace

mds/SessionMap.cc:put classes in unnamed namespace

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11450 from stiopaa1/mds_damagetable_moveClassesToCCfile

mds/DamageTable: move classes to .cc file

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11511 from stiopaa1/mds_mdsmap_addConstToMemFunctions

mds/MDSMap.h: add const to member functions

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11550 from renhwztetecs/renhw-wip-mgr-doc

doc/mgr: fix mgr how long to wait to failover

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11304 from ajarr/wip-volume-client-recovery-fixes

ceph_volume_client: fix recovery from partial auth update

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11386 from xiexingguo/xxg-wip-mgr-1008

mgr: misc minor changes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11507 from ceph/wip-17579

build/ops: deb: add python dependencies where needed

Reviewed-by: Alfredo Deza <adeza@redhat.com>

build/ops: deb: add python dependencies where needed

this reverts 48f4576 and f140a8c

Fixes: http://tracker.ceph.com/issues/17579
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

build/ops: deb: put depends on separate lines

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11557 from xiexingguo/xxg-wip-bluestore-1019

os/bluestore: kill overlay related options

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11516 from david-z/wip-add-force-ro-ceph-string

common/ceph_string: add ceph string constants for CEPH_SESSION_FORCE_RO

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11552 from tchaikov/wip-fix-enc

include/object: pass "snapid_t&" to bound_encode()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11434 from batrick/mds-cmd-deprecated

Flag deprecated mds commands and omit deprecated mon commands in help output

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11486 from batrick/client-gid

client: get gid from MonClient

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11373 from ukernel/wip-mds-recall-warning

mds: fix false "failing to respond to cache pressure" warning

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11551 from wjwithagen/wip-wjw-freebsd-lltng

do_freebsd.sh: Do not use LTTNG on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

include/object: pass "snapid_t&" to bound_encode()

otherwise the STL container's encoders will segfault when trying to
figure out the element size: they use "denc(*(A*)nullptr, elem_size)"
to do the calculation.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/mgr: fix mgr how long to wait to failover

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

do_freebsd.sh: Do not use LLTNG on FreeBSD

  FreeBSD has Dtrace

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #11534 from ceph/wip-17607-ceph-disk

tests: ceph-disk: force debug monc = 0

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: kill overlay related options

These two options are not used anymore.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Fix broken link for caps
http://tracker.ceph.com/issues/17587
Signed-off-by: Uday udaymjl@gmail.com

Merge pull request #11512 from wjwithagen/wip-wjw-freebsd-depends

FreeBSD: update require packages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11204 from liewegas/wip-rados-ack

librados: do not request osd ack if no completed completion is set

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #11213 from liewegas/wip-osd-fastinfo

osd: add a pg _fastinfo attribute to reduce per-io metadata updates

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #11354 from liewegas/wip-rocksdb

rocksdb: update to latest, and make it the default for the mons

Merge pull request #11500 from liewegas/wip-lttng

CMakeLists.txt: enable LTTNG by default

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds/SessionMap.h: remove unneeded function

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

mds/SessionMap: add const to member functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11540 from liewegas/wip-bluestore-cow

os/bluestore: default bluestore_clone_cow=true

Merge pull request #11539 from liewegas/wip-bluestore-gc

os/bluestore: no garbage collection for uncompressed blobs

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: default bluestore_clone_cow=true

This has been stable for a long time.

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "ceph_test_objecstore: use 1TB block device"

This reverts commit 5389133ff9a7783d77796ed73893e26f34c6e5d2.

This is too slow with the bitmap allocator.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: No garbage collection for uncompressed blobs

Signed-off-by: Roushan Ali<roushan.ali@sandisk.com>

Merge pull request #11536 from liewegas/wip-fullbit-dump

mds: fix EMetaBlob::fullbit xattr dump

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11533 from ifed01/wip-kstore-rmcoll-fix

os/kstore: rmcoll fix to satisfy store_test

Reviewed-by: Sage Weil <sage@redhat.com>

mds: fix EMetaBlob::fullbit xattr dump

- show xattr names!
- bound the values

Signed-off-by: Sage Weil <sage@redhat.com>

tests: ceph-disk: force debug monc = 0

The sh function will collect both stderr and stdout and debug
will mess the json parsing.

Fixes: http://tracker.ceph.com/issues/17607
Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11525 from wjwithagen/wip-wjw-freebsd-getopt

tools/ceph-monstore-update-crush.sh: FreeBSD getopt is not compatible…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11524 from wjin/filestore_perfcounter

os/filestore: clean filestore perfcounters

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11518 from umullan/uday

doc: fix broken links

Merge pull request #11522 from wjwithagen/wip-wjw-freebsd-readable.sh

FreeBSD:test/encoding/readable.sh": fix nproc and ls -v calls

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11528 from jcsp/wip-vstart-1osd

vstart: decrease pool size if <3 OSDs

Reviewed-by: Sage Weil <sage@redhat.com>

osd/PG: add perfcounters for pg info updates

Note info updates (of any kind), fastinfo updates,
and biginfo updates.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: proper set resulting pnext to avoid returning local var

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/kstore: fix KStore rm_coll handler to behave similar to bluestore one

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

ceph-objectstore-tool: relax cur_struct_v checks

All compat pg formats can handle past interval removal
and mark-complete.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: make fastinfo apply conditionally

This way we don't have to bother deleting the fastinfo
attr, which will reduce the amount of work the
underlying store has to do.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add _fastinfo PG attr for common pg_info_t updates

For most IO operations we only update a handful of fields
in the pg_info_t structure.  However, the full struct,
when encoded, is on the order of 800 bytes.

This adds a new attribute, _fastinfo, which contains only
the most commonly updated fields.  When present, the
fastinfo fields should be overlayed on top of the full
info struct contained in the existing info attr.  If
a field outside of the "fast" set is updated, we clear
the fastinfo attribute and update the full info attr.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: pass last_writen_info to _prepare_write_info

We'll use this shortly.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: operator== for pg hit sets, history, info

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop unused args to _prepare_write_info

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: pg_stat_t ondisk_log_size == log_size

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10325 from snakeAngel2015/rgw-support-s3subuser

rgw: add suport for creating S3 type subuser of admin rest api
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

os/kstore: uniform Collection implementation with BlueStore one

Signed-off-by:Igor Fedotov <ifedotov@mirantis.com>