Merge pull request #11354 from liewegas/wip-rocksdb

rocksdb: update to latest, and make it the default for the mons

Merge pull request #11500 from liewegas/wip-lttng

CMakeLists.txt: enable LTTNG by default

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11540 from liewegas/wip-bluestore-cow

os/bluestore: default bluestore_clone_cow=true

Merge pull request #11539 from liewegas/wip-bluestore-gc

os/bluestore: no garbage collection for uncompressed blobs

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: default bluestore_clone_cow=true

This has been stable for a long time.

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "ceph_test_objecstore: use 1TB block device"

This reverts commit 5389133ff9a7783d77796ed73893e26f34c6e5d2.

This is too slow with the bitmap allocator.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: No garbage collection for uncompressed blobs

Signed-off-by: Roushan Ali<roushan.ali@sandisk.com>

Merge pull request #11536 from liewegas/wip-fullbit-dump

mds: fix EMetaBlob::fullbit xattr dump

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11533 from ifed01/wip-kstore-rmcoll-fix

os/kstore: rmcoll fix to satisfy store_test

Reviewed-by: Sage Weil <sage@redhat.com>

mds: fix EMetaBlob::fullbit xattr dump

- show xattr names!
- bound the values

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11525 from wjwithagen/wip-wjw-freebsd-getopt

tools/ceph-monstore-update-crush.sh: FreeBSD getopt is not compatible…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11524 from wjin/filestore_perfcounter

os/filestore: clean filestore perfcounters

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11518 from umullan/uday

doc: fix broken links

Merge pull request #11522 from wjwithagen/wip-wjw-freebsd-readable.sh

FreeBSD:test/encoding/readable.sh": fix nproc and ls -v calls

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11528 from jcsp/wip-vstart-1osd

vstart: decrease pool size if <3 OSDs

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: proper set resulting pnext to avoid returning local var

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/kstore: fix KStore rm_coll handler to behave similar to bluestore one

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #10325 from snakeAngel2015/rgw-support-s3subuser

rgw: add suport for creating S3 type subuser of admin rest api
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

os/kstore: uniform Collection implementation with BlueStore one

Signed-off-by:Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11481 from ddiss/cmake_rgw_deps

cmake: check WITH_RADOSGW for fcgi and expat dependencies

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart: decrease pool size if <3 OSDs

Makes it easy to do OSD=1 for e.g. doing cephfs
stuff where we don't care much about the OSDs
behaviour.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #10239 from weiqiaomiao/wqm-wip-setolh

rgw: should assign 'olh_bl" to state.attrset[RGW_ATTR_OLH_ID_TAG] instead of 'bl'
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11324 from yehudasa/wip-14589

rgw: dump objects in RGWBucket::check_object_index()
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

tools/ceph-monstore-update-crush.sh: FreeBSD getopt is not compatible, use the one from packages

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #11432 from hansbogert/patch-3

doc: Update install-ceph-gateway.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

Fixes: #17587
doc: Fixing the broken link which was refactored for authentication as part of other PR
Signed-off-by: uday udaymjl@gmail.com

os/filestore: clean filestore perfcounters

Move filestore perfcounters from objectstore to filestore itself
and change internal index name.

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

Merge pull request #11027 from liewegas/wip-denc

fast denc encoding

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

Merge remote-tracking branch 'gh/kraken'

cmake: find GIT_VER variables if there is no .git dir

find CEPH_GIT_VER & CEPH_GIT_NICE_VER if there is no
ceph/.git dir by parsing ceph/src/.git_version.

scenario is for building from source tarballs generated
by make-dist, since make-dist generates .git_version
by running make_version, and then packs .git_version
into the tarball.

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit c5c6b59f296f0ace3a6cfd939027118e82ab50a4)

Merge pull request #11502 from liewegas/wip-bluestore-keybug

os/bluestore: fix escaping of chars > 0x80

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/kstore: fix escaping of chars > 0x80 in keys

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix escaping of odd chars >0x80 in keys

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: test unprintable chars

Signed-off-by: Sage Weil <sage@redhat.com>

FreeBSD:test/encoding/readable.sh": fix nproc and ls -v calls

 Neither of these is available in FreeBSD.
 But sort -n is Posix, and does the same ATM.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #11521 from markhpc/wip-send_inline

msg/async: set ms_async_send_inline to false to improve small randread iops

Set ms_async_send_inline to false to improve small randread iops.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #11262 from jcsp/wip-17270-master

Revert "osdc: After write try merge bh."

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

11.0.2

Merge pull request #11211 from ceph/wip-no-recordlock-test

test: temporarily remove fork()ing flock tests

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Fixes: #17587
doc: Fixing the broken link which was refactored for authentication as part of other PR
Signed-off-by: uday udaymjl@gmail.com

Fixes: #17587
doc: Fixing the broken link which was refactored for authentication as part of other PR
Signed-off-by: uday udaymjl@gmail.com

Merge pull request #11440 from ceph/wip-jlayton-linkref

client: don't take extra target inode reference in ll_link

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11421 from ceph/wip-jlayton-cephfs

Small pile of random cephfs fixes and cleanup

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11515 from badone/wip-find_best_info-typos

osd: Fix typos in PG::find_best_info

Reviewed-by: xiexingguo <xie.xingguo@zte.com.cn>

osd: Fix typos in PG::find_best_info

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #11513 from badone/clear_primary_state-typo

osd: fix typo in PG::clear_primary_state

Merge pull request #11514 from xiexingguo/xxg-wip-bluestore-1015

os/bluestore: formatting nits

Reviewed-by: Sage Weil <sage@redhat.com>

osd: fix typo in PG::clear_primary_state

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

buffer: make iterator copy to ptr handle len==0

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: fix * zero placement

Back in 46522cf0d27ad7d236f61bf132c915210754a0f4 we broke the newline
placement for the * lines.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-dencoder: don't dump usage on every error

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-dencoder: add hexdump command

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do deep encode of onode

Avoid an inevitable rebuild() since we have to pass this to rocksdb as a
single memory buffer anyway.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add a 'deep' append mode

Do the copy up-front.  This is useful if we know our target buffer must
be a contiguous buffer (e.g., because it will be passed to a kv store).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: log onode component sizes

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: contiguous_appender::get_logical_offset

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: write into page-aligned buffers from rocksdb

The BlueRocksEnv uses the append(const char *, size_t) appender.  Ensure
that this data is copied into a page-aligned buffer, and that we "pad" the
write out with the remainder of the page.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add page_aligned_appender

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluefs: use append(const char*, size_t)

The bufferlist one is for internal use only.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: convert internal types to denc

Signed-off-by: Sage Weil <sage@redhat.com>

osd: denc_traits for osd_reqid_t

Signed-off-by: Sage Weil <sage@redhat.com>

msg/msg_types: denc_traits for entity_name_t

Signed-off-by: Sage Weil <sage@redhat.com>

include/interval_set: define denc_traits for interval_set<T>

Signed-off-by: Sage Weil <sage@redhat.com>

include/fs_types: conditional denc_traits for inodeno_t

Signed-off-by: Sage Weil <sage@redhat.com>

include/object: conditional denc_traits for snapid_t

Signed-off-by: Sage Weil <sage@redhat.com>

include/denc: _nohead variants for containers

Signed-off-by: Sage Weil <sage@redhat.com>

include/denc: new-style encoding framework

#include it from encoding.h so that we can make old STL container helpers
conditional on a new-style helper not being available.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: implement bufferptr::iterator

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: implement bufferlist::contiguous_appender

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: fix whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: implement ptr::end_c_str()

Signed-off-by: Sage Weil <sage@redhat.com>

include/small_encoding: rewrite lowz in terms of generic varint

Signed-off-by: Sage Weil <sage@redhat.com>

include/small_encoding: rewrite small_{encode,decode}_signed_varint in terms of varint

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add iterator copy_shallow() to ptr

If the length resides within the bufferlist's current ptr, return a ptr
to the same buffer.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: rename iterator copy() to copy_deep()

Current callers expect a deep copy; be explicit about it.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-dencoder: use encode/decode functions instead of class methods

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-dencoder: AuthMonitor::Incremental is featured

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-dencoder: PGMap[::Incremental] requires features to encode

Signed-off-by: Sage Weil <sage@redhat.com>

declare encode/decode for various types that are missing it

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11443 from wjwithagen/wip-wjw-freebsd-cmake-options

do_freebsd.sh: Set options for debug building.

Reviewed-by: Kefu Chai <kchai@redhat.com>

./do_freebsd.sh: remove pipe to get correct return value

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #11430 from wjwithagen/wip-wjw-freebsd-EventKqueue

FreeBSD/EventKqueue.{h,cc} Added code to restore events on (thread)fork

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11284 from liewegas/wip-mon-osdmap-features

mon/OSDMonitor: encode full osdmaps with features all OSDs can understand

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11401 from liewegas/wip-mgr-keys

auth: tolerate missing MGR keys during upgrade

Reviewed-by: Kefu Chai <kchai@redhat.com>

auth/cephx: do not re-request *only* the MGR key

If we request a bunch of service keys, we may not get
back a MGR key because of an in-progress upgrade.  If we
have everything we need except for just the MGR key, do
not bother re-requesting it.  Instead just continue and
we'll re-request it later when the secrets rotate.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: do not request MGR key from pre-kraken mons

If we do, the older mon will return EPERM.

Signed-off-by: Sage Weil <sage@redhat.com>

client: don't take extra target inode reference in ll_link

For the life of me, I can't figure out where this reference is ever put.
We usually take a reference like this when there is an Inode **
parameter that we'll return to the caller. ll_link doesn't have one of
those, so as best I can tell this reference is just leaked.

Zheng however says that FUSE will eventually put this reference via
ll_forget. I still don't quite get how that works, but the other callers
clearly do not handle this correctly. Change the code to only make the
fuse code take this extra reference.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

src/msg/async/EventKqueue: refactor and add test_thread_change()

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

src/msg/async/EventKqueue.cc: adjust logging levels.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

src/msg/async/EventKqueue.{h,cc} Added code to restore events on (thread)fork

According the FreeBSD man page of kqueue(), the kq-descriptors become invalid
upon fork. It looks like the same happens when a kq-handle is created and then
a thread is created.

So we keep a list of assigned events with each kq-handle and when the handle
has beccome invalid, we recreated the kq-handle and the events that go with it.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #11276 from tchaikov/wip-17400

tools/ceph_monstore_tool: update pgmap_meta also when rebuilding store.db

Tested-by: Huawen Ren <ren.huanwen@zte.com.cn>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

os/bluestore: narrow scope of local variables

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: formatting nits

Remove dupilicated semicolon, 80 chars etc.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11339 from liewegas/wip-mon-stuck

mon/PGMap: PGs can be stuck more than one thing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11499 from ceph/wip-kraken-no-git-dir

cmake: find GIT_VER variables if there is no .git dir

Reviewed-by: Alfredo Deza <adeza@redhat.com>

auth/cephx: tolerate missing rotating keys

During an upgrade, we may have a client requesting an
MGR service key but not have one in the database yet,
either because we *just* upgraded and haven't generated
one yet, or because the leader mon hasn't been upgraded
yet.

Fix this by silently tolerating a missing key as long as
one or more other service keys were present and we have
something to give to the client.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11398 from ifed01/wip-bluestore-rmcoll-fix

os/bluestore: fix remove_collection to properly detect collection e…

Reviewed-by: Sage Weil <sage@redhat.com>

cmake: find GIT_VER variables if there is no .git dir

find CEPH_GIT_VER & CEPH_GIT_NICE_VER if there is no
ceph/.git dir by parsing ceph/src/.git_version.

scenario is for building from source tarballs generated
by make-dist, since make-dist generates .git_version
by running make_version, and then packs .git_version
into the tarball.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

CMakeLists.txt: enable LTTNG by default

This makes dev builds more closely match package builds.

Signed-off-by: Sage Weil <sage@redhat.com>