tools/rados/rados.cc: fix race condition in load-gen complete callback

Fixes: #14536
Signed-off-by: Tianshan Qu <tianshan@xsky.com>

Merge pull request #7337 from clever215/rbd_nbd

rbd-nbd: Add an argument validation for nbds_max

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7385 from trociny/fix-journal-flush-commit-position

journal: flush commit position on metadata shutdown

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7382 from dillaman/wip-14510

librbd: block maintenance ops until after journal is ready

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7259 from trociny/wip-14264

librbd: API: async open and close

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7302 from dillaman/wip-14434

librbd: journal shut down flush race condition

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7299 from dillaman/wip-librbd-update-features

librbd: fix internal handling of dynamic feature updates

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7381 from ceph/wip-rgw-static-website-yehuda

RGW static website feature
 - currently disabled by default
 - partial error page handling
 - a few known failures

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Robin H. Johnson <robin.johnson@dreamhost.com>

journal: flush commit position on metadata shutdown

A crash was observed on JournalTrimmer destroy, which was called
by the Journaler after metadata shutdown. JournalTrimmer destructor
also calls flush_commit_position, but at that time metadata m_timer
is already null and it crushed there.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #7223 from liewegas/wip-collection-handles

osd: collection handles

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #7134 from liewegas/wip-fusestore

osd: implement fuse interface for ObjectStore

Reviewed-by: Samuel Just <sjust@redhat.com>

osd: use collection handle where possible

Signed-off-by: Sage Weil <sage@redhat.com>

osd: associate a CollectionHandle& with PGBackend

Make this a ref to the PG's handle because the backend is
constructed before the handle is opened.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: associate a CollectionHandle with each PG

Signed-off-by: Sage Weil <sage@redhat.com>

os/memstore: implement handle-based read methods

Signed-off-by: Sage Weil <sage@redhat.com>

os/memstore: implement open_collection

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: implement CollectionHandle based read methods

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: implement open_collection

Fix up create and remove so that they poison a Collection that
gets removed.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add CollectionHandle wrappers for all read ops

The backend can optionally implement the handle-based variant. If
it does not, the wrappers will call the legacy coll_t version.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add CollectionHandle

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: block maintenance until after journal is ready

Ops were already blocked when the requests were received via
watch/notify.  Requests are now blocked for local requests as
well.

Fixes: #14510
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge branch 'wip-rgw-static-website-yehuda'

Conflicts:
src/rgw/rgw_common.h
src/rgw/rgw_op.h
src/rgw/rgw_rest_s3.cc
src/rgw/rgw_rest_s3.h

os/FuseStore: improve readdir for omap and attr

Use new-style fuse readdir interface for omap and attr.  We aren't
particularly efficient, but we won't completely break if there are a
lot of attrs or omap entries.

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: wait for commit and flush in all updates

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: use offset for readdir

This allows us to list large directories.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/objectstore/test_fuse.sh: simple workunit test

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: add 'type' file in root dir

This also acts as a sentinel to indicate whether we have mounted.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add get_type()

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: drop filestore-specific sanity check

Just let mount() fail

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: add --op mkfs

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: simplify error message

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: 'whoami' is not required for an ObjectStore

OSD creates it, but it won't be there if we did --op mkfs

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: do not call exists() on an out-of-bounds oid

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: no-op chmod

This makes mkdir(1) happy.

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: allow object and collection creates

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: more ENOENT on getattr for non-existent items

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: hide bitwise_hash_end if bits unknown; add bitwise_hash_bits

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: present bitwise_hash_end

Note this is the *last* hash value, inclusive.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: implement collection_bits

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add collection_bits

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: respect fuse_debug

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: no need to check for fsck here

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: use myexit(1) for fsck

Signed-off-by: Sage Weil <sage@redhat.com>

osd: osd_objectstore_fuse

Expose underlying ObjectStore via fuse at $osd_data/fuse/.  Can be enabled/
disabled at runtime.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: mount an ObjectStore via fuse

Currently only read-only.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: put the OSD journal in the normal location

Also, drop the ability to run without a journal.

Signed-off-by: Sage Weil <sage@redhat.com>

[g]hobject_t: normalize operator<< result, and implement parse

- print hash in *bitwise reverse* order
- include all fields, every time
- use : delimitor for hobject_t
- use @ delimitor for ghobject_t
- escape unprintable characters <32, >=127, :, %.
  - also /, for fuse layer's benefit

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7195 from liewegas/wip-rados-bench

rados: rados bench improvements

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7269 from liewegas/wip-bluestore-btree

bluestore: use btree_map for allocator

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #6 from ifed01/wip-bluestore-btree-test

UT for interval_set implementations

Merge pull request #6869 from chuanhong-wang/chuanhong-wang-wip-14026

osd: fix OSD utilization is abnormal after data disk lost

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7201 from ErwanAliasr1/evelu-cleanup-deadstuff

cleanup: remove misc dead code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7217 from vumrao/wip-14356-vumrao

osd: FileStore: cleanup: remove obsolete option "filestore_xattr_use_omap"

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7245 from Sandy4999/wip-mon-election

mon: add an independent option for max election time

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #7253 from wido/mon-err-stuck-pg

mon: go into ERR state if multiple PGs are stuck inactive

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #7281 from xuechendi/omap_sync_assert

osd: FileStore: add error check for object_map->sync()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7309 from asheplyakov/bug-14428

osd: PG::activate(): handle unexpected cached_removed_snaps more gracefully

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7319 from chenji-kael/wip-cls-clear-pool

test/cls: clean useless pools

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7320 from yuyuyu101/kill-keyvaluestore

os/keyvaluestore: kill KeyValueStore

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7375 from rzarzynski/wip-14526

rgw: fix dangeours usage of strtok in RGWAccessControlPolicy_SWIFT

Add UT for interval_set implementations

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #7379 from branch-predictor/bp-async-fixes-0127

msg/async: bunch of fixes

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async: bunch of fixes

1. Use MSG_MORE in sendmsg to reduce number of actual sends and pass hint
   to do_sendmsg in callers.
2. Move restore_sigpipe() outside of loop in do_sendmsg().
3. restore_sigpipe when hit the error condition in do_sendmsg().
3. When sent_bytes >= outcoming_bl.length, don't swap outgoing_bl with
   bl and wait for bl to get out of scope, just clear out outgoing_bl.
4. Don't swap empty outcoming_bl with bl, always claim_append(bl).

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #7329 from fullerdj/wip-14476

librbd: remove canceled tasks from timer thread

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: fix dangeours usage of strtok in RGWAccessControlPolicy_SWIFT.

Fixes: #14526
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rbd: Add an argument-check for nbds_max in rbd-nbd

Fixes: #14482
This modification adds several checks to the argument of nbds_max in the module of rbd-nbd. The value of nbds_max is strictly limited to non-negative integer now.

Signed-off-by: Kongming Wu <wu.kongming@h3c.com>

Merge pull request #7120 from zaitcev/wip-5073-update1

rgw: fix up the use of tenant before it's available

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7366 from tchaikov/wip-cmake-snappy

cmake: check for libsnappy in default path also

cmake: check for libsnappy in default path also

* some distros install libsnappy into /usr/lib/x86_64-linux-gnu
  which is completely resonable. but this directory is not listed
  in the Findsnappy.cmake.
* also add SNAPPY_ROOT_DIR, in case user install snappy in some
  other directory.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7304 from liewegas/wip-tcmalloc

osd: pass tcmalloc env through to daemon

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #7358 from chenyehua11692/master

mds:judgment added to avoid the risk of visiting the NULL pointer

Reviewed-by: Yan, Zheng <zyan@redhat.com>

upstart/ceph-osd.conf: pass TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES through

Signed-off-by: Sage Weil <sage@redhat.com>

mds:judgment added to avoid the risk of visting the NULL pointer

Signed-off-by: chenyehua11692 chen.yehua@h3c.com

Merge pull request #7251 from tchaikov/wip-rados-manpage

doc: amend the rados.8

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7353 from renhwsky/renhw-wip-14473-2

mon: fix locking in preinit error paths

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7321 from renhwsky/renhw-wip-14473

mon: drop useless rank init assignment

Reviewed-by: Sage Weil <sage@redhat.com>

mon: drop useless rank init assignment

drop useless rank init assignment

Fixes:14508
Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #7294 from branch-predictor/bp-buffer-spinlock

common/buffer: replace RWLock with spinlocks

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: song baisen <song.baisen@zte.com.cn>

Merge pull request #7303 from cbodley/wip-tx-move

os/ObjectStore: add custom move operations for ObjectStore::Transaction

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7283 from xiexingguo/xxg-wip-14407

os/bluestore/kstore: fix nid overwritten logic

http://pulpito.ceph.com/sage-2016-01-25_07:39:08-rados-wip-sage-testing2---basic-smithi/

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7267 from liewegas/wip-fiemap

osd: drop fiemap len=0 magic

http://pulpito.ceph.com/sage-2016-01-25_07:39:08-rados-wip-sage-testing2---basic-smithi/

Reviewed-by: Sage Weil <sage@redhat.com>

mon: fix locking in preinit error paths

fix locking in preinit error paths

Fixes: #14473
Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

rados: bench: add --max-objects

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7314 from tchaikov/wip-fix-cmake

cmake: let ceph-client-debug link with tcmalloc

Merge pull request #7331 from XinzeChi/wip-RepModify-bytes-written

osd: delete useless bytes_written defined in RepModify

Reviewed-by: Kefu Chai <kchai@redhat.com>

PG::activate(): handle unexpected cached_removed_snaps more gracefully

PGPool::update(): ditto

Fixes: #14428
Backport: infernalis, hammer, firefly

Signed-off-by: Alexey Sheplyakov <asheplyakov@mirantis.com>

Merge pull request #7342 from tchaikov/wip-fix-ceph-dencoder

os/bluestore: fix bluestore_wal_transaction_t encoding test

Reviewed-by: Loic Dachary <ldachary@redhat.com>

os/bluestore: fix bluestore_wal_transaction_t encoding test

bluestore_wal_op_t::op is encoded/dumped but it is not
initialized in its ctor. so set it to 0 in its ctor. 0 is not an
valid op though, but it helps with debugging and testing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7315 from tchaikov/wip-kill-warnings

Wip kill warnings

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7284 from ceph/wip-make-remove-libedit

makefile: remove libedit from libclient.la

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd: delete useless bytes_written defined in RepModify

this would avoid calling encode function.

Signed-off-by: Xinze Chi <xinze@xsky.com>

Merge pull request #7301 from oritwas/wip-14375

rgw: fix subuser rm command failure

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rbd: remove canceled tasks from timer thread

When canceling scheduled tasks using the timer thread, TaskFinisher::cancel
does not call SafeTimer::cancel_event, so events fire anyway. Add this call.

Fixes: #14476
Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #7327 from igrcic/master

doc: Fixes a CRUSH map step take argument

Reviewed-by: John Wilkins <jowilkin@redhat.com>

doc: Fixes a CRUSH map step take argument

Signed-off-by: Ivan Grcic <igrcic@gmail.com>

os: add custom move ctor/assignment for Transaction

Signed-off-by: Casey Bodley <cbodley@redhat.com>

os: add custom move ctor/assignment for TransactionData

Signed-off-by: Casey Bodley <cbodley@redhat.com>

os: Transaction sets default values to avoid duplication in ctors

Signed-off-by: Casey Bodley <cbodley@redhat.com>

os: move Transaction ctors closer to member variables

this makes it easier to update constructors when adding/removing fields

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test: add unit test for Transaction move/copy

Signed-off-by: Casey Bodley <cbodley@redhat.com>