client: always provide the file type in stx_mode

The proposed statx interface does this, and this would allow ganesha
to get away without requesting CEPH_STATX_MODE when all it wants is
enough to instantiate a filehandle.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: remove unimplemented fields from ceph_statx

We don't provide stx_version and stx_information is currently unused.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Revert "client: move the device bitshift handling macros to Client.h"

This reverts commit 2115de04a417e6df4272fc836829b70bd6a2b97e.

This is unnecessary now that we're using dev_t's in ceph_statx.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: convert ceph_statx to use dev_t for devices

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: switch ceph_statx time fields to struct timespec

This is just much more convenient for applications that want to use
the API. We're not talking to the kernel here, so there's no real
reason we need to use fixed length fields.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: remove unneeded extra padding from ceph_statx

Holdover from the proposed kernel API that doesn't really help anything
here. If we ever want to extend this struct, we'll just append to the
end and add new want mask bits.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: reinstate clearing of *fhp in ll_create and don't allow fhp == NULL

The logic in ll_create relies on *fhp being zeroed out on entry into the
function, but that's no longer being done since commit e08210dda. Fix
this by reinstating that initialization.

Luckily, it looks like the two main callers (FUSE and nfs-ganesha)
already handle this safely, but the SyntheticClient does not. Move the
zeroing of the pointer into ll_create, and allow callers to pass in a
pointer to an uninitialized value. With this, we can remove the
initialization in the fuse client code as well.

Also, we can get rid of some unneeded NULL pointer checks if we don't
allow callers to pass in a NULL fhp pointer. As best I can tell, no one
does that currently and no comments over ceph_ll_create seem to claim
that this should be allowed.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #11069 from liewegas/wip-rocksdb

rocksdb: pull up to master (4.12 + a few patches)

Merge pull request #11079 from ifed01/wip-fix-zlib

compressor/zlib: zlib wrapper fix

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11077 from xiexingguo/xxg-wip-bluestore-0914

os/bluestore: add checks and kill unreachable code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11014 from chhabaramesh/extent_alloc

os/bluestore: make assert conditional with macro for allocator

Reviewed-by: Sage Weil <sage@redhat.com>

compressor/zlis: improve buffer allocation to avoid space waste

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

common/compressor: fix improper length value passed to zlib compressor

Signed-off-by: Igor Fedotov <ifedotovo@miranitis.com>

test/objectstore: extend compression test case to use both snappy & zlib

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11071 from dmick/wip-17267

Stop hiding errors from run-tox.sh

Reviewed-by: Alfredo Deza <adeza@redhat.com>

os/bluestore: drop result check of OP_TRY_RENAME

Because _rename() never returns -ENOENT, it only
returns -EEXIST on failure.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add sanity check for compress()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Stop hiding errors from run-tox.sh

The grep -v should have been grep, but, why even bother?
Tox doesn't output that much; just dump it, and exit with
tox's error code.

Fixes: http://tracker.ceph.com/issues/17267
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #10826 from xiexingguo/xxg-wip-fix-objecter-0823

osdc/objecter: misc fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10859 from xiexingguo/xxg-wip-fix-osd-08-25

osd/osd_type: check if pool is gone during check_new_interval()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #11059 from markhpc/wip-bluefs-wal-buffered

bluestore/BlueFS: revert direct IO for WRITER_WAL

Merge pull request #11030 from amoxic/amoxic-wip-fix-do-write-small

os/bluestore: fix offset bug in _do_write_small.

Reviewed-by: Sage Weil <sage@redhat.com>

rocksdb: pull up to master (4.12 + a few patches)

This includes Somnath's fix for log recycling.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11046 from stiopaa1/common_timer_deleteCopyConstrAndAssign

common/Timer.h: delete copy constr and assign op

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11057 from liewegas/wip-os-fuse

qa/workunits/objectstore/test_fuse.sh: make test_fuse.sh work with filestore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11064 from ilc/make-srpm

make-srpm.sh: A simple script to make the srpm for ceph.

Merge pull request #11061 from xiexingguo/xxg-wip-bs-0913

os/bluestore: fix shard_info::dump()

Reviewed-by: Sage Weil <sage@redhat.com>

make-srpm.sh: A simple script to make the srpm for ceph.

This is a simple script to make the srpm for ceph based
on make-dist.

Yes, it is short, but it will stop many people from
having to solve the same problem.

Signed-off-by: Ira Cooper <ira@redhat.com>

qa/workunits/objecstore/test_fuse.sh: make it work with filestore

Signed-off-by: Sage Weil <sage@redhat.com>

Delete unnecessary parentheses.

Signed-off-by: feng yu amoxic@126.com

Merge pull request #11016 from wido/snap-max-uid

mds: Set mds_snap_max_uid to 4294967294

Reviewed-by: John Spray <john.spray@redhat.com>

min_min_alloc_size to accommodate changing min_alloc_size

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

os/bluestore: fix dump of shard_info

which shall include the "extents" field too.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10873 from dreamhost/wip-17136

rgw: Fix Host->bucket fallback logic inversion

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10918 from tianshan/fix-bucket-decode-v

rgw: fix rgw_bucket_dir_entry decode v

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

bluestore/BlueFS: revert direct IO for WRITER_WAL

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #11040 from xiexingguo/xxg-wip-bluestore-0910

os/bluestore: fix leak of result-checking of _fsck_check_extents

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11047 from tchaikov/wip-long-live-readdir

ceph_mon: use readdir() as readdir_r() is deprecated

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #11033 from xiexingguo/xxg-wip-bluestore-0909

os/bluestore: fix potential access violation during rename

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11044 from dillaman/wip-17227

librbd: ignore partial refresh error when acquiring exclusive lock

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11037 from dillaman/wip-17254

librbd: possible deadlock if cluster connection closed after image

Reviewed-by: Xiaoxi Chen <xiaoxchen@ebay.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

ceph_mon: use readdir() as readdir_r() is deprecated

see https://lwn.net/Articles/696469/, readdir_r() is deprecated by glibc
since 2.24. so let's use readdir() instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10941 from ceph/8.31.16-fix-install-deps

install-deps: exit non-zero when we cannot match distro

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11043 from stiopaa1/osd_osd_removeUnneededReturns

osd/OSD.cc: remove unneeded returns

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/OSD.cc: remove unneeded returns

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

common/Timer.h: delete copy constr and assign op

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

librbd: ignore partial refresh error when acquiring exclusive lock

Fixes: http://tracker.ceph.com/issues/17227
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11041 from stiopaa1/osd_replicatedpg_removeClassRedeclaration

osd/ReplicatedPG: remove class redeclaration

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10952 from codeyz/dev

crush/CrushCompiler.cc:884

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10986 from stiopaa1/osd_osd_remove_repeated_searching_of_map

osd/OSD.cc: remove repeated searching of map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11000 from stiopaa1/osd_osd_removeUnneededSearchMap

osd/OSD.cc: remove unneeded searching of map

Reviewed-by: Jun Huang <hjwsm1989@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10935 from branch-predictor/bp-optimized-crush-ln

crush: don't normalize input of crush_ln iteratively

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10776 from ceph/wip-offline-split

ceph-objectstore-tool: add a way to split filestore directories offline

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10011 from songbaisen/a16

 mon: remove the redundant list swap in paxos commit_proposal

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10872 from rjfd/wip-16776

Enable builds without ceph-test subpackage

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/ReplicatedPG: remove class redeclaration

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

librbd: possible deadlock if cluster connection closed after image

Fixes: http://tracker.ceph.com/issues/17254
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: drop bluestore_blob_id_t

which is not used anymore.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: kill fill() method from bluestore_extent_ref_map_t

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add sanity check during extent loading

Otherwise there are potential extent leaks.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11034 from dillaman/wip-17251

librbd: potential seg fault when blacklisting an image client

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

os/bluestore: fix leak of result-checking of _fsck_check_extents

We shall not ignore the result of _fsck_check_extents() durint fsck().

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: using is_spanning() for spanning checking

Otherwise is_spanning() has no caller and shall be deleted.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix race condition

The empty() method also needs to be called under the protection of lock.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: simplify extent_shard_key a little

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: return onode_key to caller

The get_key_extent_shard() method shall decompose extent_shard_key
into onode_key and offset, but currently we don't fill onode_key
and return it to caller.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: make extent key generation a little faster

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix potential access violation during rename

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

install-deps: exit non-zero when we cannot match distro

Signed-off-by: Gregory Meno <gmeno@redhat.com>

Merge pull request #11001 from stiopaa1/client_barrier_moveCBlockSyncClass

client/barrier: move C_Block_Sync class to .cc

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10606 from jcsp/wip-mds-standby-trim

mds: trim null dentries proactively

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph-objectstore-tool: add a way to split filestore directories offline

Use the usual split code, and split each dir that meets the
usual split criteria.

This can be run with lower than usual split settings, to avoid costly
online splits. To make sure the directories are not merged again, use
a load merge threshold (e.g. filestore merge threshold = 1), and
adjust the split multiplier accordingly.

Fixes: http://tracker.ceph.com/issues/17220
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10985 from dillaman/wip-16921

rbd-nbd: fix kernel deadlock during teuthology testing

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

deb: Moved ceph-client-debug from ceph-base to ceph-test package

Signed-off-by: Ricardo Dias <rdias@suse.com>

rpm: Added ceph_test_package bcond flag to control the build of ceph-test package

Fixes: http://tracker.ceph.com/issues/16776
Signed-off-by: Ricardo Dias <rdias@suse.com>

cmake: Use WITH_TESTS flag to guard the build of ceph-test scripts/binaries

Signed-off-by: Ricardo Dias <rdias@suse.com>

librbd: potential seg fault when blacklisting an image client

Fixes: http://tracker.ceph.com/issues/17251
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ignore cache busy errors when shrinking an image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: invalidate cache before trimming image

Any potential writeback outside the extents of a shrunk image
would result in orphaned objects.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-nbd: mask out-of-bounds IO errors caused by image shrink

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-nbd: fix kernel deadlock during teuthology testing

Fixes: http://tracker.ceph.com/issues/16921
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rpm: Renamed bcond tests flag to make_check

Signed-off-by: Ricardo Dias <rdias@suse.com>

rpm: Moved ceph-client-debug from ceph-base to ceph-test package

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #10723 from ceph/wip-os-fuse

os/FuseStore: fix several FuseStore issues

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: Something may be wrong with _do_write_small.
  The statement "if (ep->logical_offset + b->get_blob().get_ondisk_length() <= offset)" which try to judge if the writing offset is in the current Blob may be wrong.

Signed-off-by: feng yu amoxic@126.com

Merge pull request #9767 from dx9/wip-safe-io

safe_io: Improve portability by replacing loff_t type usage with off_t.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11028 from ceph/wip-17450

cmake: set ARM_CRC_FLAGS from the CRC test rather than ARM_NEON_FLAGS

Reviewed-by: Kefu Chai <kchai@redhat.com>

Set ARM_CRC_FLAGS from the CRC test rather than ARM_NEON_FLAGS

Fixes: http://tracker.ceph.com/issues/17250
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #10302 from stiopaa1/msg_asyncconn_addConstToMemFun

msg/AsyncConnection: add const to mem functions

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10114 from stiopaa1/msg_asyncmess_get_proto_version_addConst

msg/AsyncMessenger: add const to function

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11015 from tchaikov/wip-optional-py3

cmake: make py3 a nice-to-have

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge branch 'rgwfs-fix-rgw-lookup' of https://github.com/cmgitdream/ceph into cmgitdream-rgwfs-fix-rgw-lookup

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

osd/OSD.cc: remove unneeded searching of map

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

client/barrier: move C_Block_Sync class to .cc

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

msg/AsyncMessenger: add const to function

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

msg/AsyncConnection: add const to mem functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11003 from ceph/wip-cmake

cmake: include(SIMDExt) in src/CMakeLists.txt

Merge pull request #11023 from liewegas/wip-bluefs-extent-append

os/bluestore/BlueFS: only extend extent on same bdev

os/bluestore/BlueFS: only extend extent on same bdev

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10822 from Yan-waller/yj-wip-rgwfile-0823

rgw_file: fix return value signedness (rgw_readdir)

verified: f23