Merge pull request #10313 from ceph/objclass-perm-feedback

Objclass perm feedback

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10250 from gregsfortytwo/wip-sessionmap-committing

mds: Return "committing" rather than "committed" member in get_committing

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10346 from tchaikov/wip-make-check

test: ceph-disk.sh do not kill all daemons

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Revert "run-make-check.sh: run tests in two steps"

This reverts commit cd0dfc305d3861b9baf332a1cc13f1c716d8017b.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "cmake: label some tests with "Racing""

This reverts commit 3a21d0040a7a96aefcf0f4876ee4006c4225019c.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: ceph-disk.sh: do not kill all daemons

this causes mysterious "racing" issues when running ceph-helper tests in
parallel.

Fixes: http://tracker.ceph.com/issues/16729
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10317 from wjwithagen/wip-wjw-fix-dl-lib

Cmake: fix using CMAKE_DL_LIBS instead of dl

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10147 from SUSE/wip-16598

rpm: move mount.ceph from ceph-base to ceph-common and add symlink in /sbin for SUSE

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com

Merge pull request #9411 from ceph/wip-rocksdb-perf

os/RocksDBStore: use effective Get API instead of iterator api

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10130 from yonghengdexin735/wip-zzz-add-

src/common/buffer.cc fix judgment for lseek

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10197 from songbaisen/b0

mon: write fsid use the right return value

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9291 from trociny/wip-15715

rbd-nbd does not properly handle resize notifications

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9878 from neurodrone/rbd_api_shrink_check_on_resize

librbd: add explicit shrink check while resizing images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9955 from ukernel/wip-mds-snap-failover

mds: snap failover fixes

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10283 from ceph/wip-cmake

cmake changes

Reviewed-by: Ali Maredia <amaredia@redhat.com>

build/ops: build mount.ceph and mount.fuse.ceph as client binaries

Signed-off-by: Nathan Cutler <ncutler@suse.com>

os/bluestore: use p2 macros to simplify bit-allocator block alignment

Mark's comments:

This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests.

Closes #10253

os/bluestore: require block_size to be power of 2 aligned

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
os/bluestore: use ISP2 macro for zone/span size checking

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix potential uninitialized nid of onode

The _zero() process may implicitly create a new onode,
thus we shall call _assign_nid() to initialize the nid
properly. And if the onode already has one, _assign_nid()
does nothing.

So it is proper to call _assign_nid() here under any case.

Mark's comments:

This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests.

Closes #10236

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
os/bluestore: check against we don't overflow

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
os/bluestore: try to reap as many collections as we can

So if there is one collection getting contiguously stucking,
we don't abort at the same point each time.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
os/bluestore: make device size of BitFreelistManager is block-size aligned

Otherwise if we try to set past-eof blocks as allocated durint create(),
the call to _xor() will trigger the firing of the following assert:

   assert((length & block_mask) == length);

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: end scope of std::hex properly; convert csum error to EIO

Mark's comments:

This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests.

Closes #10225

os/bluestore: end scope of std::hex properly

To avoid side-effects by accident.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
os/bluestore: convert csum error to EIO

The verify_csum() method either returns -1 or -EOPNOTSUPP, which
is not very proper and difficult for user understanding.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
os/bluestore: assert lextent is shared

Otherwise we are risking of accessing violation.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
os/bluestore: drop duplicated assignment of result code

These two methods never fail actually.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
os/bluestore: improve _do_read() a little

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
os/bluestore: assert decoding of shard of key to be successful

Otherwise we are risking of acessing null pointer.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rpm: move mount.ceph from ceph-base to ceph-common

Ceph clients use mount.ceph to mount CephFS filesystems, and
ceph-base is not expected to be installed on client systems.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: create mount.ceph symlink in /sbin (SUSE only)

Fixes: http://tracker.ceph.com/issues/16598
Signed-off-by: Nathan Cutler <ncutler@suse.com>

os/bluestore: revert preferred csum behavior

This passes "ceph_test_objectstore --gtest_filter=*/2".
This restores 4K random read performance to previous levels when objects
are were previously written out using large IOs (4MB in this case):

pre-patch: 26MB/s
post-pated: 610MB/s

Closes #10320

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Bluestore: Fixed a Bluestore crash

A bluestore race condition is been fixed by protecting txc structures
within _txc_state_poc with collection lock.

Mark's comments:

This fixes segfaults during random write tests with bluestore.
This passes "ceph_test_objectstore --gtest_filter=*/2".
This may introduce a small performance regresion, though there is enough
noise in the results to make it inconclusive.

Closes #10220

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

Cmake: fix using CMAKE_DL_LIBS instead of dl

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

cmake: only allow up to 1 hour for a ceph test

quote from
https://cmake.org/cmake/help/v3.0/prop_test/TIMEOUT.html?highlight=timeout

> If it exceeds that the test process will be killed and ctest will move
> to the next test.

this helps us to identify test hang.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: restructure src/CMakeLists.txt in a more hierarchical way

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not pass --disable-pip-version-check if not supported

on older versions of pip, this option is not supported, and
--disable-pip-version-check is implied with --no-index. so no need to
use them when --no-index is passed to pip.

this partially reverts 395f2c5

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove duplicated src from ceph_rgw_jsonparser

they are included by rgw_a as well. and ceph_rgw_jsonparser is linked
against rgw_a.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link libcommon against $CRYPTO_LIBS

as ceph_crypt.cc is using the symbols in it, and libcommon contains
ceph_crypt.cc.

Signed-off-by: Kefu Chai <kchai@redhat.com>

run-make-check.sh: run tests in two steps

this is a workaround of the timeout found in jenkins. currently three
tests are found timeout, and they are labeld with "Racing" and
"LongRunning". so, to workaround this issue, we run the tests in two
phases:

1. run the racing tests with -j1
2. run the non-racing tests with -jN

if we all all tests with -j1, the total test time is 2683.57 sec

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: label some tests with "Racing"

two tests timesout for unknown reasons, so label them with
"Racing" and "LongRunning" labels.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: no need to depend run-tox-ceph-disk on test

run-tox-ceph-disk and run-tox-ceph-detect-init are already added as
test, so "ctest" and "make {test,check}" will run them without extra
settings.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add a "tests" target to build tests

please note "make test" is used by cmake to run tests, so we cannot just
repurpose it to *build* them.

* AddCephTest.cmake: depends on "tests"
* CMakeLists.txt: let "check" depend on "tests"
* src/CMakeLists.txt: update the run-tox tests
* run-make-check.sh: use "make tests" and "ctest" instead of "make check"
* ceph-detect-init/CMakeLists.txt: let "tests" depend on
    "ceph-detect-init"
* ceph-disk/CMakeLists.txt: let "tests" depend on "ceph-disk"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10301 from cbodley/wip-rgw-meta-stack-wakeup

rgw: RGWMetaSyncCR holds refs to stacks instead of crs

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

release: release notes update for objclass-perms

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

osd: use static method and simplify return

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

rgw: RGWMetaSyncCR holds refs to stacks for wakeup

because RGWCoroutine::wakeup() calls RGWCoroutinesStack::wakeup(), the
stack must also stay alive

Fixes: http://tracker.ceph.com/issues/16666
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10306 from tchaikov/wip-no-mktemp-p

qa/workunits/cephtool/test.sh: s/TMPDIR/TEMP_DIR/

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #9972 from ceph/objclass-perm

osd: object class loading and execution permissions

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9980 from gaowanlong/split_out_handle_pg_scrub

osd: small cleanups

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10126 from dx9/wip-fcntl-warns

test/libcephfs: fix gcc sys/fcntl.h warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10166 from wjwithagen/wip-wjw-cmake-test_rados_tool.sh

test_rados_tool.sh: Make script work under ctest

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9782 from Ved-vampir/zlib_cleanup

compressor: zlib compressor plugin  cleanup

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: run_seed_to_range.sh: check existance of a directory using [ -d "$dir" ]

sadly, sh evalutes `[ -d ]` to true. as it takes "-d" as a non-empty
string as true.

this fixes following failure
```
2016-07-12T23:22:02.839 INFO:teuthology.orchestra.run.mira084.stderr:cp:
missing destination file operand after ‘.’
2016-07-12T23:22:02.839 INFO:teuthology.orchestra.run.mira084.stderr:Try
'cp --help' for more information.
```
see
http://pulpito.ceph.com/kchai-2016-07-12_23:09:35-rados-wip-kefu-testing2---basic-mira/311334/

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: s/TMPDIR/TEMP_DIR/

this fixes the test failure of
```
2016-07-12T23:29:40.935
INFO:tasks.workunit.client.0.mira101.stderr:/home/ubuntu/cephtest/workunit.client.0/cephtool/test.sh:
line 153: /CEPH_WATCH_9445: Permission denied
```
see
http://pulpito.ceph.com/kchai-2016-07-12_23:09:35-rados-wip-kefu-testing2---basic-mira/311333/

it's a regression introduced by e5c262b

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: use mktemp $TEMP_DIR/XXX instead

mktemp -p is not supported on FreeBSD

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10131 from badone/wip-peering-doc-fixes

doc: peering.rst, fix typo

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10292 from badone/wip-perf-counters-doc-fixes

doc: perf_counters.rst fix trivial typo

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test: ceph-detect-init/run-tox.sh: run it from any path

this follows the pattern in ceph-disk. this enables us to run
ceph-detect-init/run-tox.sh from the ${CMAKE_BINARY_DIRECTORY}
as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

client: fix MetaRequest::set_other_inode()

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: close directory's snapdir when deleting directory

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: invalidate snap inodes after removing snapshot

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: fix incorrect "unconnected snaprealm xxx" warning

If a snaprealm has no child/parent snaprelam, and the snaprealm inode
is not in the cache while client reconnects. The snaprealm does not
get properly removed from MDCache::reconnected_snaplrealm. This causes
incorrect "unconnected snaprealm xxx" warning

Signed-off-by: Yan, Zheng <zyan@redhat.com>

qa/workunits/fs: fix expect_failure function in test scripts

The origin expect_failure function return 0 regardness of command's
return value.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: make sure snapflush is sent before normal cap message

MDS does null snapflush when it receives normal cap message. So client
must send snapflush first.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: unify cap flush and snapcap flush

This patch includes following changes
- assign flush tid to snapcap flush
- remove session's flushing_capsnaps list. add inode with snapcap
  flushes to session's flushing_caps list instead.
- when reconnecting to MDS, re-send one inode's snapcap flushes and
  cap flushes at the same time.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: handle partly purged directory

For a snapshoted direcotry whose snaprealm parents are being opened,
MDS does not know if the directory is purgeable. So MDS can't skip
committing dirfrags of the directory. But if the direcotry is purgeale,
some dirfrags could have already been deleted during MDS failover.
Committing them could return -ENOENT.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: do files recovery after processing cap flushes

File recovery may update inode and trigger inode COW. MDS relies on
client caps to setup CInode::client_need_snapflush. But for a given
client, the reconnected caps may not include the flushing caps.
(Before MDS failover, client released and flushed some caps at the
same time. When MDS recovers, client re-send the cap flush and send
cap reconnect to the MDS.) This may cause later snapflush to get
dropped.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: combine MDCache::{reconnected_caps,cap_imports_dirty}

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: remove CEPH_LOCK_IFLOCKL from cinode_lock_info

Currently we don't support dirty CEPH_CAP_FLOCK_EXCL

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: rebuild the internal states that tracking pending snapflush

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: using cap_reconnect_t to track cap recconect

Previous commit extended cap_reconnect_t to include snapflush
related information. This information are needed in various
places

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: add 'follows' of first pending snapflush to cap reconnect message

This helps the recovering MDS to reconstruct the internal states that
tracking pending snapflush.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: journal snap inodes that need flush when expiring log segment

Treat snap inodes that need flush in the same way as open files.
When MDS recovers, this make sure that journal replay bring snap
inodes that need flush into the cache

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: properly update client_snap_caps when splitting snap inode

update the new snap inode's client_snap_caps according to the old
snap inode.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #9491 from Yan-waller/yj-wip-fixbuildpushop

osd/ReplicatedBackend: add sanity check during build_push_op()

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #9510 from renhwztetecs/renhw-wip-common

common: unknown hash type of judgment modification

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9729 from stiopaa1/common_workqueue_addMove

common/WorkQueue: add std move

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9770 from tanghaodong25/submit-transaction-perfcounter

kv/RocksDB: add perfcounter for submit_transaction_sync operation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10274 from yehudasa/wip-16665

rgw: collect skips a specific coroutine stack

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10187 from dreamhost/wip-16618

rgw: fix multi-delete query param parsing.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10289 from jdurgin/wip-googletest-centos

fix the rpm build for centos

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10271 from linuxbox2/rgw-rename

rgw_file: fix rename cases and unify unlink
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8618 from linuxbox2/rgw-file-setattr

rgw: file setattr
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw_file: implement rgw_setattr

Introduce a new RGWSetattrs RGWOp descendant, to create or replace
sets of attrs on buckets or objects.

This version of the change uses the standard RGWRADOS::set_attrs op
(we want attribute changes to (e.g.) sync with other changes).

Previous versions of this changed incorrectly masked the values
of st->st_ino in RGWFileHandle::stat(), now fixed.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #10284 from linuxbox2/rgwfile_write_guard

rgw_file: refuse partial, out-of-order writes
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10258 from ukernel/wip-16655

ceph-fuse: link to libtcmalloc or jemalloc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10168 from batrick/client-inode-des

client: move Inode specific cleanup to destructor

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10169 from batrick/client-inode-cons

client: note order of member init in cons

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10265 from tchaikov/wip-ceph-fs-warnings

include/ceph_fs.h: guard `#define CEPH_SETATTR_*` with #ifndef

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10142 from batrick/i16042

mds: move Finisher to unlocked shutdown

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10176 from renhwztetecs/renhw-wip-mon-mds

mon/mds: add err info when load_metadata is abnormal

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10192 from batrick/mon-minor

mon: use clearer code structure

Reviewed-by: songbaisen <song.baisen@zte.com.cn>

Merge pull request #10200 from renhwztetecs/renhw-wip-mds-journal

osdc/mds: fixup pos parameter in the journaler

Reviewed-by: John Spray <john.spray@redhat.com>

doc: peering.rst, fix typo

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

doc: perf_counters.rst fix trivial typo

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #9933 from chhabaramesh/master

kv: In memory keyvalue db implementation

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

install-deps: Get the latest virtualenv in a temporary folder
to work around a bug in old virtualenv

Signed-off-by: Oleh Prypin <oleh@pryp.in>

googletest: avoid installing any pieces of it

This makes rpmbuild happy.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

rbd: add rbd_resize2 for allow_shrink option

Merge pull request #10286 from ceph/wip_leftover_with_mds

Removes remaining reference to WITH_MDS

Reviewed-by: Ali Maredia <amaredia@redhat.com>

rgw: collect skips a specific coroutine stack

Fixes: http://tracker.ceph.com/issues/16665
Instead of drain_all_but() that specifies number of stacks to leave behind,
added drain_all_but_stack() that has a specific stack specified. This is needed
so that we don't call wakeup() through lease_cr->go_down() on a cr stack that
was already collected.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Not all references to WITH_MDS were removed (see PR #10186). This
removes the final one (dependency of vstart on ceph-mds).

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #10120 from oritwas/bz16412

rgw: remove bucket index objects when deleting the bucket

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10161 from ceph/wip-dang-cmake

CMake - stop pip checking for updates

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10170 from ceph/wip-pg-log-errors-11

include write error codes in the pg log

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge branch 'googletest'

Update Ceph to use Google Test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

objclass: implement 'osd class default list'

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

rgw_file: refuse partial, out-of-order writes

A single file object may be opened only once per gateway
instance, and writes to that object must be complete, and in-order.
Enforce this.

If an invalid write is seen, deletes the current write transaction.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #10282 from ceph/wip-cmake-cxx-flags

cmake: fix the FTBFS introduced by dc8b3ba

Reviewed-by: Casey Bodley <cbodley@redhat.com>