os/memstore: return empty list if offset overflows

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/memstore: fix wrong blocksize for statfs

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #7404 from sahithi-rv/Mutex

rbd_fuse: replaced pthread_mutex*() functions with ceph Mutex wrapper

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6131 from rzarzynski/wip-admincreds-in-s3-keystone

rgw: support admin credentials in S3-related Keystone authentication.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: support admin credentials in S3-related Keystone authentication.

Fixes: #13302
Backport: infernalis, hammer
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #7298 from racpatel/wip-doc-rac2

doc :- fixing image in section ERASURE CODING

Reviewed-by: Loic Dachary <ldachary@redhat.com>

cmake: musl libc does not implement backtrace. Added feature check and fallback to libexecinfo.

Signed-off-by: John Coyle <dx9err@gmail.com>

Merge pull request #7264 from ceph/wip-doc-RGW

doc: rgw: port changes from downstream to upstream

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7244 from mikulely/fix-authtool-doc

doc: remove redundant space in ceph-authtool/monmaptool doc

doc: Updated the RPM section for RHEL and CentOS

Fixes: #10245
Signed-off-by: Aron Gunn <ritz_303@yahoo.com>

Merge pull request #7027 from mathslinux/fix-s3-csharp-doc

doc: fix S3 C# example

Merge pull request #7000 from zhouyuan/fix_cache_tier_doc

doc: fix misleading configuration guide on cache tiering

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6930 from wido/rgw-key-docs

doc: use 'ceph auth get-or-create' for creating RGW keyring

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7384 from wido/mirroring

doc: script and guidelines for mirroring Ceph

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #7435 from xiexingguo/xxg-wip-14561

osd: BlueStore: fix null pointer access

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7271 from somnathr/wip-transaction-delete

osd: avoid heap allocations for ObjectStore::Transaction

http://pulpito.ceph.com/sage-2016-01-28_12:50:51-rados-wip-sage-testing3---basic-mira/

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6440 from stiopaa1/scrubbingInfo

mon: warn if pg(s) not scrubbed

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #7075 from shun-s/shun-fix

global: do not start two daemons with a single pid-file

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7111 from chenji-kael/master

osd: FileStore: print file name before osd assert if read file failed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7268 from ifed01/wip-14400

common: snappy decompressor may assert when handling segmented input bufferlist

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7276 from caibo2014/fix-messager-null

mon/MonClient: avoid null pointer error when configured incorrectly

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7330 from ifed01/wip-pool-stat-cleanup

librados: remove duplicate definitions for rados pool_stat_t and cluster_stat_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7340 from ceph/wip-portable-time

common: ake ceph_time clocks work under BSD

Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge pull request #7341 from songbaisen/song7

mon: consider the pool size when setting pool crush rule

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7355 from yuanjingkai/master

os/fs: fix io_getevents argument

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7348 from xiexingguo/xxg-wip-14484

osd: BlueStore: fix space leak

http://pulpito.ceph.com/sage-2016-01-27_18:39:48-rados-wip-sage-testing---basic-smithi/

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7368 from somnathr/wip-jemalloc-build

build: allow jemalloc enabled build with rocksdb-static

http://pulpito.ceph.com/sage-2016-01-27_18:39:48-rados-wip-sage-testing---basic-smithi/

Reviewed-by: Sage Weil <sage@redhat.com>

BlueStore: fix null pointer access

Fixes: #14561
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rbd_fuse:replaced pthread_mutex*() functions with ceph Mutex wrapper

Signed-off-by: Sahithi R V <tansy.rv@gmail.com>

Merge remote-tracking branch 'origin/greg-fs-testing-126'

Merge remote-tracking branch 'origin/jewel'

Merge pull request #7312 from jcsp/wip-14365

mds, client: fix locking around handle_conf_change

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #7297 from ukernel/jewel-13903

client: properly trim unlinked inode

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7262 from ukernel/jewel-14380

Be more careful about the "mds setmap" command to prevent breakage.

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #7369 from mauri/fix-message-typo

rbd: fix error message typo

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7387 from theanalyst/fix/gitignore

remove unused src/messages/#MOSDOp.h# file

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7345 from x11562/rbd_snap_rename

rbd: add "--dest-snap" optional for snap rename

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7393 from tchaikov/wip-cmake

fix the autotools and cmake build (the new fusestore needs libfuse)

Reviewed-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #7403 from rzarzynski/wip-rgw-dumpconlen-cosmetics

rgw: cosmetic improvements in dump_content_length()

rgw: cosmetic improvements in dump_content_length().

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

ceph-objectstore-tool: disable fuse support if !HAVE_LIBFUSE

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: disable FuseStore if !HAVE_LIBFUSE

Signed-off-by: Kefu Chai <kchai@redhat.com>

make: disable FuseStore if !WITH_FUSE

* configure.ac: "#define HAVE_LIBFUSE" if libfuse is found
* os/Makefile.am: do not compile FuseStore.cc if libfuse is not found

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link against libfuse if HAVE_LIBFUSE

* use ${FUSE_LIBRARIES} instead of "fuse" when referencing libfuse.
* do not compile FuseStore.cc if !HAVE_LIBFUSE.
* CMakeLists.txt: s/HAVE_FUSE/HAVE_LIBFUSE/, to match with the one
  defined for config.h.in.cmake, otherwise we need two vars:
  HAVE_LIBFUSE, and HAVE_FUSE

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove the copyright of FindSnappy.cmake

as we've basically removed all the HyperTable code.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Script and Guidelines for mirroring Ceph

This commit introduces a script which admins can use to mirror
the Ceph packages to their local systems.

With this script they can easily sync from a mirror local to them.

The README explains user on how and when to sync the sources and
how they can become a official mirror for Ceph.

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #6875 from dx9/wip-cmake-ccache

cmake: support ccache via a WITH_CCACHE build option

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd: simplify snap rename arguments

Signed-off-by: Na Xie <xie.na@h3c.com>

Merge pull request #7337 from clever215/rbd_nbd

rbd-nbd: Add an argument validation for nbds_max

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7385 from trociny/fix-journal-flush-commit-position

journal: flush commit position on metadata shutdown

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7382 from dillaman/wip-14510

librbd: block maintenance ops until after journal is ready

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

OSD: Deleting transaction object right after applying transaction

Presently, the transaction object is been deleted by the Finisher
thread asynchronously. In heavy load scenario specially if we
unleash journal throttle more, we are seeing high memory usage by
the OSDs because of this. In this new scheme, with the help of
move semantics transaction objects will be deleted synchronously
from the filestore worker threads. We are seeing very much
controllable memory growth now as well as ~3 to 4% cpu usage
benefit because of some reduction of 'new' , 'delete' calls.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

Merge pull request #7259 from trociny/wip-14264

librbd: API: async open and close

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7302 from dillaman/wip-14434

librbd: journal shut down flush race condition

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7299 from dillaman/wip-librbd-update-features

librbd: fix internal handling of dynamic feature updates

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

gitignore: ignore backup files from editors etc.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

remove unused source file

This file was possibly accidentally committed in the tree, thanks
@BranchPredictor for bringing this up in irc

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #7381 from ceph/wip-rgw-static-website-yehuda

RGW static website feature
 - currently disabled by default
 - partial error page handling
 - a few known failures

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Robin H. Johnson <robin.johnson@dreamhost.com>

journal: flush commit position on metadata shutdown

A crash was observed on JournalTrimmer destroy, which was called
by the Journaler after metadata shutdown. JournalTrimmer destructor
also calls flush_commit_position, but at that time metadata m_timer
is already null and it crushed there.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #7223 from liewegas/wip-collection-handles

osd: collection handles

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #7134 from liewegas/wip-fusestore

osd: implement fuse interface for ObjectStore

Reviewed-by: Samuel Just <sjust@redhat.com>

osd: use collection handle where possible

Signed-off-by: Sage Weil <sage@redhat.com>

osd: associate a CollectionHandle& with PGBackend

Make this a ref to the PG's handle because the backend is
constructed before the handle is opened.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: associate a CollectionHandle with each PG

Signed-off-by: Sage Weil <sage@redhat.com>

os/memstore: implement handle-based read methods

Signed-off-by: Sage Weil <sage@redhat.com>

os/memstore: implement open_collection

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: implement CollectionHandle based read methods

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: implement open_collection

Fix up create and remove so that they poison a Collection that
gets removed.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add CollectionHandle wrappers for all read ops

The backend can optionally implement the handle-based variant. If
it does not, the wrappers will call the legacy coll_t version.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add CollectionHandle

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: block maintenance until after journal is ready

Ops were already blocked when the requests were received via
watch/notify.  Requests are now blocked for local requests as
well.

Fixes: #14510
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge branch 'wip-rgw-static-website-yehuda'

Conflicts:
src/rgw/rgw_common.h
src/rgw/rgw_op.h
src/rgw/rgw_rest_s3.cc
src/rgw/rgw_rest_s3.h

os/FuseStore: improve readdir for omap and attr

Use new-style fuse readdir interface for omap and attr.  We aren't
particularly efficient, but we won't completely break if there are a
lot of attrs or omap entries.

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: wait for commit and flush in all updates

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: use offset for readdir

This allows us to list large directories.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/objectstore/test_fuse.sh: simple workunit test

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: add 'type' file in root dir

This also acts as a sentinel to indicate whether we have mounted.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add get_type()

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: drop filestore-specific sanity check

Just let mount() fail

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: add --op mkfs

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: simplify error message

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: 'whoami' is not required for an ObjectStore

OSD creates it, but it won't be there if we did --op mkfs

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: do not call exists() on an out-of-bounds oid

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: no-op chmod

This makes mkdir(1) happy.

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: allow object and collection creates

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: more ENOENT on getattr for non-existent items

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: hide bitwise_hash_end if bits unknown; add bitwise_hash_bits

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: present bitwise_hash_end

Note this is the *last* hash value, inclusive.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: implement collection_bits

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add collection_bits

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: respect fuse_debug

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: no need to check for fsck here

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: use myexit(1) for fsck

Signed-off-by: Sage Weil <sage@redhat.com>

osd: osd_objectstore_fuse

Expose underlying ObjectStore via fuse at $osd_data/fuse/.  Can be enabled/
disabled at runtime.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: mount an ObjectStore via fuse

Currently only read-only.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: put the OSD journal in the normal location

Also, drop the ability to run without a journal.

Signed-off-by: Sage Weil <sage@redhat.com>

[g]hobject_t: normalize operator<< result, and implement parse

- print hash in *bitwise reverse* order
- include all fields, every time
- use : delimitor for hobject_t
- use @ delimitor for ghobject_t
- escape unprintable characters <32, >=127, :, %.
  - also /, for fuse layer's benefit

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7195 from liewegas/wip-rados-bench

rados: rados bench improvements

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7269 from liewegas/wip-bluestore-btree

bluestore: use btree_map for allocator

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #6 from ifed01/wip-bluestore-btree-test

UT for interval_set implementations