osd/command tell: check pgid at the right time

Run a command: "ceph tell osd.0 query", and it outputs the following
error message:
  Error EINVAL: unrecognized command! [{"prefix": "query"}]

In fact, the command "query" exists, but it requires one more input
parameter "pgid", therefore the following error information will be
more appropriate:
  Error EINVAL: no pgid specified

In this patch, we check the parameter pgid after recognizing a command
(like "query" or "list_missing"), rather than before it.

Signed-off-by: Javeme <javaloveme@gmail.com>

Merge pull request #11512 from wjwithagen/wip-wjw-freebsd-depends

FreeBSD: update require packages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11204 from liewegas/wip-rados-ack

librados: do not request osd ack if no completed completion is set

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #11213 from liewegas/wip-osd-fastinfo

osd: add a pg _fastinfo attribute to reduce per-io metadata updates

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #11354 from liewegas/wip-rocksdb

rocksdb: update to latest, and make it the default for the mons

Merge pull request #11500 from liewegas/wip-lttng

CMakeLists.txt: enable LTTNG by default

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11540 from liewegas/wip-bluestore-cow

os/bluestore: default bluestore_clone_cow=true

Merge pull request #11539 from liewegas/wip-bluestore-gc

os/bluestore: no garbage collection for uncompressed blobs

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: default bluestore_clone_cow=true

This has been stable for a long time.

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "ceph_test_objecstore: use 1TB block device"

This reverts commit 5389133ff9a7783d77796ed73893e26f34c6e5d2.

This is too slow with the bitmap allocator.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: No garbage collection for uncompressed blobs

Signed-off-by: Roushan Ali<roushan.ali@sandisk.com>

Merge pull request #11536 from liewegas/wip-fullbit-dump

mds: fix EMetaBlob::fullbit xattr dump

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11533 from ifed01/wip-kstore-rmcoll-fix

os/kstore: rmcoll fix to satisfy store_test

Reviewed-by: Sage Weil <sage@redhat.com>

mds: fix EMetaBlob::fullbit xattr dump

- show xattr names!
- bound the values

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11525 from wjwithagen/wip-wjw-freebsd-getopt

tools/ceph-monstore-update-crush.sh: FreeBSD getopt is not compatible…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11524 from wjin/filestore_perfcounter

os/filestore: clean filestore perfcounters

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11518 from umullan/uday

doc: fix broken links

Merge pull request #11522 from wjwithagen/wip-wjw-freebsd-readable.sh

FreeBSD:test/encoding/readable.sh": fix nproc and ls -v calls

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11528 from jcsp/wip-vstart-1osd

vstart: decrease pool size if <3 OSDs

Reviewed-by: Sage Weil <sage@redhat.com>

osd/PG: add perfcounters for pg info updates

Note info updates (of any kind), fastinfo updates,
and biginfo updates.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: proper set resulting pnext to avoid returning local var

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/kstore: fix KStore rm_coll handler to behave similar to bluestore one

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

ceph-objectstore-tool: relax cur_struct_v checks

All compat pg formats can handle past interval removal
and mark-complete.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: make fastinfo apply conditionally

This way we don't have to bother deleting the fastinfo
attr, which will reduce the amount of work the
underlying store has to do.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add _fastinfo PG attr for common pg_info_t updates

For most IO operations we only update a handful of fields
in the pg_info_t structure.  However, the full struct,
when encoded, is on the order of 800 bytes.

This adds a new attribute, _fastinfo, which contains only
the most commonly updated fields.  When present, the
fastinfo fields should be overlayed on top of the full
info struct contained in the existing info attr.  If
a field outside of the "fast" set is updated, we clear
the fastinfo attribute and update the full info attr.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: pass last_writen_info to _prepare_write_info

We'll use this shortly.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: operator== for pg hit sets, history, info

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop unused args to _prepare_write_info

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: pg_stat_t ondisk_log_size == log_size

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10325 from snakeAngel2015/rgw-support-s3subuser

rgw: add suport for creating S3 type subuser of admin rest api
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

os/kstore: uniform Collection implementation with BlueStore one

Signed-off-by:Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11481 from ddiss/cmake_rgw_deps

cmake: check WITH_RADOSGW for fcgi and expat dependencies

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart: decrease pool size if <3 OSDs

Makes it easy to do OSD=1 for e.g. doing cephfs
stuff where we don't care much about the OSDs
behaviour.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #10239 from weiqiaomiao/wqm-wip-setolh

rgw: should assign 'olh_bl" to state.attrset[RGW_ATTR_OLH_ID_TAG] instead of 'bl'
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11324 from yehudasa/wip-14589

rgw: dump objects in RGWBucket::check_object_index()
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

tools/ceph-monstore-update-crush.sh: FreeBSD getopt is not compatible, use the one from packages

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #11432 from hansbogert/patch-3

doc: Update install-ceph-gateway.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

Fixes: #17587
doc: Fixing the broken link which was refactored for authentication as part of other PR
Signed-off-by: uday udaymjl@gmail.com

os/filestore: clean filestore perfcounters

Move filestore perfcounters from objectstore to filestore itself
and change internal index name.

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

Merge pull request #11027 from liewegas/wip-denc

fast denc encoding

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

Merge remote-tracking branch 'gh/kraken'

cmake: find GIT_VER variables if there is no .git dir

find CEPH_GIT_VER & CEPH_GIT_NICE_VER if there is no
ceph/.git dir by parsing ceph/src/.git_version.

scenario is for building from source tarballs generated
by make-dist, since make-dist generates .git_version
by running make_version, and then packs .git_version
into the tarball.

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit c5c6b59f296f0ace3a6cfd939027118e82ab50a4)

Merge pull request #11502 from liewegas/wip-bluestore-keybug

os/bluestore: fix escaping of chars > 0x80

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/kstore: fix escaping of chars > 0x80 in keys

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix escaping of odd chars >0x80 in keys

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: test unprintable chars

Signed-off-by: Sage Weil <sage@redhat.com>

FreeBSD:test/encoding/readable.sh": fix nproc and ls -v calls

 Neither of these is available in FreeBSD.
 But sort -n is Posix, and does the same ATM.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #11521 from markhpc/wip-send_inline

msg/async: set ms_async_send_inline to false to improve small randread iops

Set ms_async_send_inline to false to improve small randread iops.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #11262 from jcsp/wip-17270-master

Revert "osdc: After write try merge bh."

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

11.0.2

Merge pull request #11211 from ceph/wip-no-recordlock-test

test: temporarily remove fork()ing flock tests

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Fixes: #17587
doc: Fixing the broken link which was refactored for authentication as part of other PR
Signed-off-by: uday udaymjl@gmail.com

Fixes: #17587
doc: Fixing the broken link which was refactored for authentication as part of other PR
Signed-off-by: uday udaymjl@gmail.com

Merge pull request #11440 from ceph/wip-jlayton-linkref

client: don't take extra target inode reference in ll_link

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11421 from ceph/wip-jlayton-cephfs

Small pile of random cephfs fixes and cleanup

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11515 from badone/wip-find_best_info-typos

osd: Fix typos in PG::find_best_info

Reviewed-by: xiexingguo <xie.xingguo@zte.com.cn>

osd: Fix typos in PG::find_best_info

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #11513 from badone/clear_primary_state-typo

osd: fix typo in PG::clear_primary_state

Merge pull request #11514 from xiexingguo/xxg-wip-bluestore-1015

os/bluestore: formatting nits

Reviewed-by: Sage Weil <sage@redhat.com>

osd: fix typo in PG::clear_primary_state

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

FreeBSD: update require packages

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

buffer: make iterator copy to ptr handle len==0

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: fix * zero placement

Back in 46522cf0d27ad7d236f61bf132c915210754a0f4 we broke the newline
placement for the * lines.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-dencoder: don't dump usage on every error

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-dencoder: add hexdump command

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do deep encode of onode

Avoid an inevitable rebuild() since we have to pass this to rocksdb as a
single memory buffer anyway.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add a 'deep' append mode

Do the copy up-front.  This is useful if we know our target buffer must
be a contiguous buffer (e.g., because it will be passed to a kv store).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: log onode component sizes

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: contiguous_appender::get_logical_offset

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: write into page-aligned buffers from rocksdb

The BlueRocksEnv uses the append(const char *, size_t) appender.  Ensure
that this data is copied into a page-aligned buffer, and that we "pad" the
write out with the remainder of the page.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add page_aligned_appender

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluefs: use append(const char*, size_t)

The bufferlist one is for internal use only.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: convert internal types to denc

Signed-off-by: Sage Weil <sage@redhat.com>

osd: denc_traits for osd_reqid_t

Signed-off-by: Sage Weil <sage@redhat.com>

msg/msg_types: denc_traits for entity_name_t

Signed-off-by: Sage Weil <sage@redhat.com>

include/interval_set: define denc_traits for interval_set<T>

Signed-off-by: Sage Weil <sage@redhat.com>

include/fs_types: conditional denc_traits for inodeno_t

Signed-off-by: Sage Weil <sage@redhat.com>

include/object: conditional denc_traits for snapid_t

Signed-off-by: Sage Weil <sage@redhat.com>

include/denc: _nohead variants for containers

Signed-off-by: Sage Weil <sage@redhat.com>

include/denc: new-style encoding framework

#include it from encoding.h so that we can make old STL container helpers
conditional on a new-style helper not being available.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: implement bufferptr::iterator

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: implement bufferlist::contiguous_appender

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: fix whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: implement ptr::end_c_str()

Signed-off-by: Sage Weil <sage@redhat.com>

include/small_encoding: rewrite lowz in terms of generic varint

Signed-off-by: Sage Weil <sage@redhat.com>

include/small_encoding: rewrite small_{encode,decode}_signed_varint in terms of varint

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add iterator copy_shallow() to ptr

If the length resides within the bufferlist's current ptr, return a ptr
to the same buffer.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: rename iterator copy() to copy_deep()

Current callers expect a deep copy; be explicit about it.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-dencoder: use encode/decode functions instead of class methods

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-dencoder: AuthMonitor::Incremental is featured

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-dencoder: PGMap[::Incremental] requires features to encode

Signed-off-by: Sage Weil <sage@redhat.com>

declare encode/decode for various types that are missing it

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11443 from wjwithagen/wip-wjw-freebsd-cmake-options

do_freebsd.sh: Set options for debug building.

Reviewed-by: Kefu Chai <kchai@redhat.com>

./do_freebsd.sh: remove pipe to get correct return value

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #11430 from wjwithagen/wip-wjw-freebsd-EventKqueue

FreeBSD/EventKqueue.{h,cc} Added code to restore events on (thread)fork

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11284 from liewegas/wip-mon-osdmap-features

mon/OSDMonitor: encode full osdmaps with features all OSDs can understand

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11401 from liewegas/wip-mgr-keys

auth: tolerate missing MGR keys during upgrade

Reviewed-by: Kefu Chai <kchai@redhat.com>

auth/cephx: do not re-request *only* the MGR key

If we request a bunch of service keys, we may not get
back a MGR key because of an in-progress upgrade.  If we
have everything we need except for just the MGR key, do
not bother re-requesting it.  Instead just continue and
we'll re-request it later when the secrets rotate.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: do not request MGR key from pre-kraken mons

If we do, the older mon will return EPERM.

Signed-off-by: Sage Weil <sage@redhat.com>