OSD: Deleting transaction object right after applying transaction

Presently, the transaction object is been deleted by the Finisher
thread asynchronously. In heavy load scenario specially if we
unleash journal throttle more, we are seeing high memory usage by
the OSDs because of this. In this new scheme, with the help of
move semantics transaction objects will be deleted synchronously
from the filestore worker threads. We are seeing very much
controllable memory growth now as well as ~3 to 4% cpu usage
benefit because of some reduction of 'new' , 'delete' calls.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

Merge pull request #7381 from ceph/wip-rgw-static-website-yehuda

RGW static website feature
 - currently disabled by default
 - partial error page handling
 - a few known failures

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Robin H. Johnson <robin.johnson@dreamhost.com>

Merge pull request #7223 from liewegas/wip-collection-handles

osd: collection handles

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #7134 from liewegas/wip-fusestore

osd: implement fuse interface for ObjectStore

Reviewed-by: Samuel Just <sjust@redhat.com>

osd: use collection handle where possible

Signed-off-by: Sage Weil <sage@redhat.com>

osd: associate a CollectionHandle& with PGBackend

Make this a ref to the PG's handle because the backend is
constructed before the handle is opened.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: associate a CollectionHandle with each PG

Signed-off-by: Sage Weil <sage@redhat.com>

os/memstore: implement handle-based read methods

Signed-off-by: Sage Weil <sage@redhat.com>

os/memstore: implement open_collection

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: implement CollectionHandle based read methods

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: implement open_collection

Fix up create and remove so that they poison a Collection that
gets removed.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add CollectionHandle wrappers for all read ops

The backend can optionally implement the handle-based variant. If
it does not, the wrappers will call the legacy coll_t version.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add CollectionHandle

Signed-off-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-rgw-static-website-yehuda'

Conflicts:
src/rgw/rgw_common.h
src/rgw/rgw_op.h
src/rgw/rgw_rest_s3.cc
src/rgw/rgw_rest_s3.h

os/FuseStore: improve readdir for omap and attr

Use new-style fuse readdir interface for omap and attr.  We aren't
particularly efficient, but we won't completely break if there are a
lot of attrs or omap entries.

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: wait for commit and flush in all updates

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: use offset for readdir

This allows us to list large directories.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/objectstore/test_fuse.sh: simple workunit test

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: add 'type' file in root dir

This also acts as a sentinel to indicate whether we have mounted.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add get_type()

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: drop filestore-specific sanity check

Just let mount() fail

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: add --op mkfs

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: simplify error message

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: 'whoami' is not required for an ObjectStore

OSD creates it, but it won't be there if we did --op mkfs

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: do not call exists() on an out-of-bounds oid

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: no-op chmod

This makes mkdir(1) happy.

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: allow object and collection creates

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: more ENOENT on getattr for non-existent items

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: hide bitwise_hash_end if bits unknown; add bitwise_hash_bits

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: present bitwise_hash_end

Note this is the *last* hash value, inclusive.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: implement collection_bits

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add collection_bits

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: respect fuse_debug

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: no need to check for fsck here

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: use myexit(1) for fsck

Signed-off-by: Sage Weil <sage@redhat.com>

osd: osd_objectstore_fuse

Expose underlying ObjectStore via fuse at $osd_data/fuse/.  Can be enabled/
disabled at runtime.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: mount an ObjectStore via fuse

Currently only read-only.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: put the OSD journal in the normal location

Also, drop the ability to run without a journal.

Signed-off-by: Sage Weil <sage@redhat.com>

[g]hobject_t: normalize operator<< result, and implement parse

- print hash in *bitwise reverse* order
- include all fields, every time
- use : delimitor for hobject_t
- use @ delimitor for ghobject_t
- escape unprintable characters <32, >=127, :, %.
  - also /, for fuse layer's benefit

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7195 from liewegas/wip-rados-bench

rados: rados bench improvements

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7269 from liewegas/wip-bluestore-btree

bluestore: use btree_map for allocator

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #6 from ifed01/wip-bluestore-btree-test

UT for interval_set implementations

Merge pull request #6869 from chuanhong-wang/chuanhong-wang-wip-14026

osd: fix OSD utilization is abnormal after data disk lost

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7201 from ErwanAliasr1/evelu-cleanup-deadstuff

cleanup: remove misc dead code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7217 from vumrao/wip-14356-vumrao

osd: FileStore: cleanup: remove obsolete option "filestore_xattr_use_omap"

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7245 from Sandy4999/wip-mon-election

mon: add an independent option for max election time

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #7253 from wido/mon-err-stuck-pg

mon: go into ERR state if multiple PGs are stuck inactive

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #7281 from xuechendi/omap_sync_assert

osd: FileStore: add error check for object_map->sync()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7309 from asheplyakov/bug-14428

osd: PG::activate(): handle unexpected cached_removed_snaps more gracefully

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7319 from chenji-kael/wip-cls-clear-pool

test/cls: clean useless pools

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7320 from yuyuyu101/kill-keyvaluestore

os/keyvaluestore: kill KeyValueStore

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7375 from rzarzynski/wip-14526

rgw: fix dangeours usage of strtok in RGWAccessControlPolicy_SWIFT

Add UT for interval_set implementations

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #7379 from branch-predictor/bp-async-fixes-0127

msg/async: bunch of fixes

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async: bunch of fixes

1. Use MSG_MORE in sendmsg to reduce number of actual sends and pass hint
   to do_sendmsg in callers.
2. Move restore_sigpipe() outside of loop in do_sendmsg().
3. restore_sigpipe when hit the error condition in do_sendmsg().
3. When sent_bytes >= outcoming_bl.length, don't swap outgoing_bl with
   bl and wait for bl to get out of scope, just clear out outgoing_bl.
4. Don't swap empty outcoming_bl with bl, always claim_append(bl).

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #7329 from fullerdj/wip-14476

librbd: remove canceled tasks from timer thread

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: fix dangeours usage of strtok in RGWAccessControlPolicy_SWIFT.

Fixes: #14526
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #7120 from zaitcev/wip-5073-update1

rgw: fix up the use of tenant before it's available

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7366 from tchaikov/wip-cmake-snappy

cmake: check for libsnappy in default path also

cmake: check for libsnappy in default path also

* some distros install libsnappy into /usr/lib/x86_64-linux-gnu
  which is completely resonable. but this directory is not listed
  in the Findsnappy.cmake.
* also add SNAPPY_ROOT_DIR, in case user install snappy in some
  other directory.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7304 from liewegas/wip-tcmalloc

osd: pass tcmalloc env through to daemon

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #7358 from chenyehua11692/master

mds:judgment added to avoid the risk of visiting the NULL pointer

Reviewed-by: Yan, Zheng <zyan@redhat.com>

upstart/ceph-osd.conf: pass TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES through

Signed-off-by: Sage Weil <sage@redhat.com>

mds:judgment added to avoid the risk of visting the NULL pointer

Signed-off-by: chenyehua11692 chen.yehua@h3c.com

Merge pull request #7251 from tchaikov/wip-rados-manpage

doc: amend the rados.8

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7353 from renhwsky/renhw-wip-14473-2

mon: fix locking in preinit error paths

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7321 from renhwsky/renhw-wip-14473

mon: drop useless rank init assignment

Reviewed-by: Sage Weil <sage@redhat.com>

mon: drop useless rank init assignment

drop useless rank init assignment

Fixes:14508
Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #7294 from branch-predictor/bp-buffer-spinlock

common/buffer: replace RWLock with spinlocks

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: song baisen <song.baisen@zte.com.cn>

Merge pull request #7303 from cbodley/wip-tx-move

os/ObjectStore: add custom move operations for ObjectStore::Transaction

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7283 from xiexingguo/xxg-wip-14407

os/bluestore/kstore: fix nid overwritten logic

http://pulpito.ceph.com/sage-2016-01-25_07:39:08-rados-wip-sage-testing2---basic-smithi/

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7267 from liewegas/wip-fiemap

osd: drop fiemap len=0 magic

http://pulpito.ceph.com/sage-2016-01-25_07:39:08-rados-wip-sage-testing2---basic-smithi/

Reviewed-by: Sage Weil <sage@redhat.com>

mon: fix locking in preinit error paths

fix locking in preinit error paths

Fixes: #14473
Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

rados: bench: add --max-objects

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7314 from tchaikov/wip-fix-cmake

cmake: let ceph-client-debug link with tcmalloc

Merge pull request #7331 from XinzeChi/wip-RepModify-bytes-written

osd: delete useless bytes_written defined in RepModify

Reviewed-by: Kefu Chai <kchai@redhat.com>

PG::activate(): handle unexpected cached_removed_snaps more gracefully

PGPool::update(): ditto

Fixes: #14428
Backport: infernalis, hammer, firefly

Signed-off-by: Alexey Sheplyakov <asheplyakov@mirantis.com>

Merge pull request #7342 from tchaikov/wip-fix-ceph-dencoder

os/bluestore: fix bluestore_wal_transaction_t encoding test

Reviewed-by: Loic Dachary <ldachary@redhat.com>

os/bluestore: fix bluestore_wal_transaction_t encoding test

bluestore_wal_op_t::op is encoded/dumped but it is not
initialized in its ctor. so set it to 0 in its ctor. 0 is not an
valid op though, but it helps with debugging and testing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7315 from tchaikov/wip-kill-warnings

Wip kill warnings

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7284 from ceph/wip-make-remove-libedit

makefile: remove libedit from libclient.la

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd: delete useless bytes_written defined in RepModify

this would avoid calling encode function.

Signed-off-by: Xinze Chi <xinze@xsky.com>

Merge pull request #7301 from oritwas/wip-14375

rgw: fix subuser rm command failure

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rbd: remove canceled tasks from timer thread

When canceling scheduled tasks using the timer thread, TaskFinisher::cancel
does not call SafeTimer::cancel_event, so events fire anyway. Add this call.

Fixes: #14476
Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #7327 from igrcic/master

doc: Fixes a CRUSH map step take argument

Reviewed-by: John Wilkins <jowilkin@redhat.com>

doc: Fixes a CRUSH map step take argument

Signed-off-by: Ivan Grcic <igrcic@gmail.com>

os: add custom move ctor/assignment for Transaction

Signed-off-by: Casey Bodley <cbodley@redhat.com>

os: add custom move ctor/assignment for TransactionData

Signed-off-by: Casey Bodley <cbodley@redhat.com>

os: Transaction sets default values to avoid duplication in ctors

Signed-off-by: Casey Bodley <cbodley@redhat.com>

os: move Transaction ctors closer to member variables

this makes it easier to update constructors when adding/removing fields

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test: add unit test for Transaction move/copy

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cleanup: remove obsolete option "filestore_xattr_use_omap"

This patch does cleanup for option "filestore_xattr_use_omap",
as this option was removed in #7408.

Fixes: #14356
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>

KeyValueStore: Kill this

Signed-off-by: Haomai Wang <haomai@xsky.com>

test/cls ：clean useless pools

After do ./ceph_test_cls_log ,pools remains
need be destroyed.

Signed-off-by: Ji Chen <insomnia@139.com>

rgw: fix up the use of tenant before it's available

A big problem that cropped up with the multi-tenancy as it
went in was that it used tenant to initialize bucket names
before tenant was actually available by parsing a token.
Thus, although multi-tenancy worked, accessing anything across
tenants was actually impossible. This patch fixes that oversight.
Unfortunately, a new method postauth_init() was needed for it.

We also fix small bugs:

 - validate_tenant_name() used a proper-looking C++ but was
   incorrect. We simply use C for now, which may be not as
   pretty but is correct.

 - We now make a distinction between empty tenant syntax that
   means "the logged-in tenant" ("bucket") and the syntax
   that means "the legacy compatibility tenant with no name
   (":bucket").

 - We make rgw_make_bucket_entry_name() more robust against
   empty inputs. It's not used for work, but produces odd-looking
   debugging logs.

Signed-off-by: Pete Zaitcev <zaitcev@redhat.com>

rgw: add a configurable to enable/disable static website

rgw_enable_static_website

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix null dereference

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't abort on op_ret < 0

We'll later have op->execute() return a status

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7317 from dmick/master

configure.ac: make "--with-librocksdb-static" default to 'check'

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

configure.ac: make "--with-librocksdb-static" default to 'check'

Fixes: #14463
Signed-off-by: Dan Mick <dan.mick@redhat.com>