tools/cephfs: add pg_files command

Sometimes users know that particular data pool PGs
have been damaged, and they would like to scan
their files to work out which ones might have
been affected.

Fixes: http://tracker.ceph.com/issues/17249
Signed-off-by: John Spray <john.spray@redhat.com>

libcephfs: take ref when constructed with CephContext

Previously, if you used ceph_create_with_context, then
it would try and destroy your context when you called
ceph_release.  Bad behaviour: caller created the
context, caller should decide when to destroy it.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #11541 from stiopaa1/mds_sessionmap_addConst

mds/SessionMap: add const to member functions

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

unittest_bluestore_types: add random put_ref test case

This appeared to fail under load, but works fine here.
:/

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: fix gc tests

In ef8989ce7417034aa715de4fe83d1f0363c044f4 we disabled
gc for non-compressed blobs.  So mark these test ones as
compressed.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11438 from trociny/wip-17488

rbd-nbd: support disabling auto-exclusive lock transition logic

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11546 from umullan/uday

doc: Fix broken link for caps

Merge pull request #11558 from ifed01/wip-rmcoll-more-fixes

os/memstore, os/filestore: fix store_test's to satisfy rm_coll behavi…

Reviewed-by: Sage Weil <sage@redhat.com>

os/memstore, os/filestore: fix store_test's to satisfy rm_coll behavior expected in store_test

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11207 from yehudasa/wip-17372

rgw: get_zonegroup() uses "default" zonegroup if empty
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10919 from pritha-srivastava/wip-17076

rgw: Replacing '+' with "%20" in canonical uri for s3 v4 auth.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11316 from stiopaa1/mds_sessionMapUnnamedNamespace

mds/SessionMap.cc:put classes in unnamed namespace

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11450 from stiopaa1/mds_damagetable_moveClassesToCCfile

mds/DamageTable: move classes to .cc file

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11511 from stiopaa1/mds_mdsmap_addConstToMemFunctions

mds/MDSMap.h: add const to member functions

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11550 from renhwztetecs/renhw-wip-mgr-doc

doc/mgr: fix mgr how long to wait to failover

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11304 from ajarr/wip-volume-client-recovery-fixes

ceph_volume_client: fix recovery from partial auth update

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11386 from xiexingguo/xxg-wip-mgr-1008

mgr: misc minor changes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11507 from ceph/wip-17579

build/ops: deb: add python dependencies where needed

Reviewed-by: Alfredo Deza <adeza@redhat.com>

build/ops: deb: add python dependencies where needed

this reverts 48f4576 and f140a8c

Fixes: http://tracker.ceph.com/issues/17579
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

build/ops: deb: put depends on separate lines

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11557 from xiexingguo/xxg-wip-bluestore-1019

os/bluestore: kill overlay related options

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11516 from david-z/wip-add-force-ro-ceph-string

common/ceph_string: add ceph string constants for CEPH_SESSION_FORCE_RO

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11552 from tchaikov/wip-fix-enc

include/object: pass "snapid_t&" to bound_encode()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11434 from batrick/mds-cmd-deprecated

Flag deprecated mds commands and omit deprecated mon commands in help output

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11486 from batrick/client-gid

client: get gid from MonClient

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11373 from ukernel/wip-mds-recall-warning

mds: fix false "failing to respond to cache pressure" warning

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11551 from wjwithagen/wip-wjw-freebsd-lltng

do_freebsd.sh: Do not use LTTNG on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

include/object: pass "snapid_t&" to bound_encode()

otherwise the STL container's encoders will segfault when trying to
figure out the element size: they use "denc(*(A*)nullptr, elem_size)"
to do the calculation.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/mgr: fix mgr how long to wait to failover

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

do_freebsd.sh: Do not use LLTNG on FreeBSD

  FreeBSD has Dtrace

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #11534 from ceph/wip-17607-ceph-disk

tests: ceph-disk: force debug monc = 0

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: kill overlay related options

These two options are not used anymore.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Fix broken link for caps
http://tracker.ceph.com/issues/17587
Signed-off-by: Uday udaymjl@gmail.com

Merge pull request #11512 from wjwithagen/wip-wjw-freebsd-depends

FreeBSD: update require packages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11204 from liewegas/wip-rados-ack

librados: do not request osd ack if no completed completion is set

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #11213 from liewegas/wip-osd-fastinfo

osd: add a pg _fastinfo attribute to reduce per-io metadata updates

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #11354 from liewegas/wip-rocksdb

rocksdb: update to latest, and make it the default for the mons

Merge pull request #11500 from liewegas/wip-lttng

CMakeLists.txt: enable LTTNG by default

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds/SessionMap: add const to member functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11540 from liewegas/wip-bluestore-cow

os/bluestore: default bluestore_clone_cow=true

Merge pull request #11539 from liewegas/wip-bluestore-gc

os/bluestore: no garbage collection for uncompressed blobs

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: default bluestore_clone_cow=true

This has been stable for a long time.

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "ceph_test_objecstore: use 1TB block device"

This reverts commit 5389133ff9a7783d77796ed73893e26f34c6e5d2.

This is too slow with the bitmap allocator.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: No garbage collection for uncompressed blobs

Signed-off-by: Roushan Ali<roushan.ali@sandisk.com>

Merge pull request #11536 from liewegas/wip-fullbit-dump

mds: fix EMetaBlob::fullbit xattr dump

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11533 from ifed01/wip-kstore-rmcoll-fix

os/kstore: rmcoll fix to satisfy store_test

Reviewed-by: Sage Weil <sage@redhat.com>

mds: fix EMetaBlob::fullbit xattr dump

- show xattr names!
- bound the values

Signed-off-by: Sage Weil <sage@redhat.com>

tests: ceph-disk: force debug monc = 0

The sh function will collect both stderr and stdout and debug
will mess the json parsing.

Fixes: http://tracker.ceph.com/issues/17607
Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11525 from wjwithagen/wip-wjw-freebsd-getopt

tools/ceph-monstore-update-crush.sh: FreeBSD getopt is not compatible…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11524 from wjin/filestore_perfcounter

os/filestore: clean filestore perfcounters

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11518 from umullan/uday

doc: fix broken links

Merge pull request #11522 from wjwithagen/wip-wjw-freebsd-readable.sh

FreeBSD:test/encoding/readable.sh": fix nproc and ls -v calls

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11528 from jcsp/wip-vstart-1osd

vstart: decrease pool size if <3 OSDs

Reviewed-by: Sage Weil <sage@redhat.com>

osd/PG: add perfcounters for pg info updates

Note info updates (of any kind), fastinfo updates,
and biginfo updates.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: proper set resulting pnext to avoid returning local var

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/kstore: fix KStore rm_coll handler to behave similar to bluestore one

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

ceph-objectstore-tool: relax cur_struct_v checks

All compat pg formats can handle past interval removal
and mark-complete.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: make fastinfo apply conditionally

This way we don't have to bother deleting the fastinfo
attr, which will reduce the amount of work the
underlying store has to do.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add _fastinfo PG attr for common pg_info_t updates

For most IO operations we only update a handful of fields
in the pg_info_t structure.  However, the full struct,
when encoded, is on the order of 800 bytes.

This adds a new attribute, _fastinfo, which contains only
the most commonly updated fields.  When present, the
fastinfo fields should be overlayed on top of the full
info struct contained in the existing info attr.  If
a field outside of the "fast" set is updated, we clear
the fastinfo attribute and update the full info attr.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: pass last_writen_info to _prepare_write_info

We'll use this shortly.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: operator== for pg hit sets, history, info

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop unused args to _prepare_write_info

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: pg_stat_t ondisk_log_size == log_size

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10325 from snakeAngel2015/rgw-support-s3subuser

rgw: add suport for creating S3 type subuser of admin rest api
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

os/kstore: uniform Collection implementation with BlueStore one

Signed-off-by:Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11481 from ddiss/cmake_rgw_deps

cmake: check WITH_RADOSGW for fcgi and expat dependencies

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart: decrease pool size if <3 OSDs

Makes it easy to do OSD=1 for e.g. doing cephfs
stuff where we don't care much about the OSDs
behaviour.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #10239 from weiqiaomiao/wqm-wip-setolh

rgw: should assign 'olh_bl" to state.attrset[RGW_ATTR_OLH_ID_TAG] instead of 'bl'
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11324 from yehudasa/wip-14589

rgw: dump objects in RGWBucket::check_object_index()
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

tools/ceph-monstore-update-crush.sh: FreeBSD getopt is not compatible, use the one from packages

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #11432 from hansbogert/patch-3

doc: Update install-ceph-gateway.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

Fixes: #17587
doc: Fixing the broken link which was refactored for authentication as part of other PR
Signed-off-by: uday udaymjl@gmail.com

os/filestore: clean filestore perfcounters

Move filestore perfcounters from objectstore to filestore itself
and change internal index name.

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

Merge pull request #11027 from liewegas/wip-denc

fast denc encoding

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

Merge remote-tracking branch 'gh/kraken'

cmake: find GIT_VER variables if there is no .git dir

find CEPH_GIT_VER & CEPH_GIT_NICE_VER if there is no
ceph/.git dir by parsing ceph/src/.git_version.

scenario is for building from source tarballs generated
by make-dist, since make-dist generates .git_version
by running make_version, and then packs .git_version
into the tarball.

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit c5c6b59f296f0ace3a6cfd939027118e82ab50a4)

Merge pull request #11502 from liewegas/wip-bluestore-keybug

os/bluestore: fix escaping of chars > 0x80

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/kstore: fix escaping of chars > 0x80 in keys

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix escaping of odd chars >0x80 in keys

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: test unprintable chars

Signed-off-by: Sage Weil <sage@redhat.com>

FreeBSD:test/encoding/readable.sh": fix nproc and ls -v calls

 Neither of these is available in FreeBSD.
 But sort -n is Posix, and does the same ATM.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #11521 from markhpc/wip-send_inline

msg/async: set ms_async_send_inline to false to improve small randread iops

Set ms_async_send_inline to false to improve small randread iops.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

mgr: shutdown finisher in a graceful way

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/MgrMonitor: make period of sending digests configurable

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/MgrMonitor: break loop on catching specified standby

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/MgrMonitor: unify updating map code path

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/MgrStandby: do not ignore result of init() of monc

Which can fail due to missing keyring.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/MgrStandby: use mgr_beacon_period to send beacon

Fixes: http://tracker.ceph.com/issues/17455
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11262 from jcsp/wip-17270-master

Revert "osdc: After write try merge bh."

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

11.0.2

Merge pull request #11211 from ceph/wip-no-recordlock-test

test: temporarily remove fork()ing flock tests

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Fixes: #17587
doc: Fixing the broken link which was refactored for authentication as part of other PR
Signed-off-by: uday udaymjl@gmail.com

Fixes: #17587
doc: Fixing the broken link which was refactored for authentication as part of other PR
Signed-off-by: uday udaymjl@gmail.com

Merge pull request #11440 from ceph/wip-jlayton-linkref

client: don't take extra target inode reference in ll_link

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11421 from ceph/wip-jlayton-cephfs

Small pile of random cephfs fixes and cleanup

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

common/ceph_string: add ceph string constants for CEPH_SESSION_FORCE_RO

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

Merge pull request #11515 from badone/wip-find_best_info-typos

osd: Fix typos in PG::find_best_info

Reviewed-by: xiexingguo <xie.xingguo@zte.com.cn>

osd: Fix typos in PG::find_best_info

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #11513 from badone/clear_primary_state-typo

osd: fix typo in PG::clear_primary_state