Merge pull request #7396 from tchaikov/wip-14537

mon: compact full epochs also

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7402 from liewegas/wip-rados-bench

rados: make rados bench metadata backward compatible

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7418 from renhwsky/renhw-wip-monmapmonitor-cleanup

mon: some cleanup in MonmapMonitor.cc

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #7420 from mikulely/fix-osd-mon-exit-issue

osd, mon: fix exit issue

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7430 from xiexingguo/xxg-wip-kstore

osd: kstore: fix decode logic and add flush

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7432 from branch-predictor/bp-fix-read-bench

rados: fix wrong op/object sizes in rand/seq bench

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7436 from liewegas/wip-bash

tests: qa/workunits/objectstore/test_fuse.sh: no bashism

Merge pull request #7404 from sahithi-rv/Mutex

rbd_fuse: replaced pthread_mutex*() functions with ceph Mutex wrapper

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6131 from rzarzynski/wip-admincreds-in-s3-keystone

rgw: support admin credentials in S3-related Keystone authentication.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: support admin credentials in S3-related Keystone authentication.

Fixes: #13302
Backport: infernalis, hammer
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #7298 from racpatel/wip-doc-rac2

doc :- fixing image in section ERASURE CODING

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph_osd.cc: fix unreachable flush call

The calling chain for generic_server_usage():

  generic_server_usage()
    ->generic_usage(true)
    ->exit(1)
  cout.flush()

any statements after generic_server_usage() would not be reached,so we
need to flush cout in generic_usage().

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

cmake: musl libc does not implement backtrace. Added feature check and fallback to libexecinfo.

Signed-off-by: John Coyle <dx9err@gmail.com>

Merge pull request #7264 from ceph/wip-doc-RGW

doc: rgw: port changes from downstream to upstream

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7244 from mikulely/fix-authtool-doc

doc: remove redundant space in ceph-authtool/monmaptool doc

doc: Updated the RPM section for RHEL and CentOS

Fixes: #10245
Signed-off-by: Aron Gunn <ritz_303@yahoo.com>

Merge pull request #7027 from mathslinux/fix-s3-csharp-doc

doc: fix S3 C# example

Merge pull request #7000 from zhouyuan/fix_cache_tier_doc

doc: fix misleading configuration guide on cache tiering

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6930 from wido/rgw-key-docs

doc: use 'ceph auth get-or-create' for creating RGW keyring

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7384 from wido/mirroring

doc: script and guidelines for mirroring Ceph

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #7435 from xiexingguo/xxg-wip-14561

osd: BlueStore: fix null pointer access

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7271 from somnathr/wip-transaction-delete

osd: avoid heap allocations for ObjectStore::Transaction

http://pulpito.ceph.com/sage-2016-01-28_12:50:51-rados-wip-sage-testing3---basic-mira/

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6440 from stiopaa1/scrubbingInfo

mon: warn if pg(s) not scrubbed

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #7075 from shun-s/shun-fix

global: do not start two daemons with a single pid-file

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7111 from chenji-kael/master

osd: FileStore: print file name before osd assert if read file failed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7268 from ifed01/wip-14400

common: snappy decompressor may assert when handling segmented input bufferlist

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7276 from caibo2014/fix-messager-null

mon/MonClient: avoid null pointer error when configured incorrectly

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7330 from ifed01/wip-pool-stat-cleanup

librados: remove duplicate definitions for rados pool_stat_t and cluster_stat_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7340 from ceph/wip-portable-time

common: ake ceph_time clocks work under BSD

Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge pull request #7341 from songbaisen/song7

mon: consider the pool size when setting pool crush rule

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7355 from yuanjingkai/master

os/fs: fix io_getevents argument

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7348 from xiexingguo/xxg-wip-14484

osd: BlueStore: fix space leak

http://pulpito.ceph.com/sage-2016-01-27_18:39:48-rados-wip-sage-testing---basic-smithi/

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7368 from somnathr/wip-jemalloc-build

build: allow jemalloc enabled build with rocksdb-static

http://pulpito.ceph.com/sage-2016-01-27_18:39:48-rados-wip-sage-testing---basic-smithi/

Reviewed-by: Sage Weil <sage@redhat.com>

BlueStore: fix null pointer access

Fixes: #14561
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/workunits/objectstore/test_fuse.sh: no bashism

Signed-off-by: Sage Weil <sage@redhat.com>

BlueStore: fix typo

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rbd_fuse:replaced pthread_mutex*() functions with ceph Mutex wrapper

Signed-off-by: Sahithi R V <tansy.rv@gmail.com>

kstore: flush before we really start a clone

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

kstore: flush before we really start a truncate

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

BlueStore: add fast check against empty list

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

kstore: add fast check against empty list

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

tools/rados: fix wrong op/object sizes in rand/seq bench

Metadata contain correct information about last bench write run, yet
these are ignored in subsequent bench rand/seq runs, resulting in
wrong perf numbers and failures when not using --no-verify.

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

kstore: simplify open_collection logic a bit

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

kstore: fix unmatched type of decode

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph_osd.cc/ceph_mon.cc: cleanup unreachable exit call

The calling chain of usage() is:

  usage()
    ->generic_server_usage()
      ->exit(1)
  exit(0)

so the exit(0) after usage() would not be reached.

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

mon:some cleanup in MonmapMonitor.h

delete unused "get_monmap(MonMap &m)"

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

mon:some cleanup in MonmapMonitor.cc

remove the processing of  ”mon add“ && “mon remove” in the preprocess_command;
delete unused "get_monmap(MonMap &m)"

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #7312 from jcsp/wip-14365

mds, client: fix locking around handle_conf_change

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge remote-tracking branch 'origin/greg-fs-testing-126'

Merge remote-tracking branch 'origin/jewel'

Merge pull request #7297 from ukernel/jewel-13903

client: properly trim unlinked inode

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7262 from ukernel/jewel-14380

Be more careful about the "mds setmap" command to prevent breakage.

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #7369 from mauri/fix-message-typo

rbd: fix error message typo

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7387 from theanalyst/fix/gitignore

remove unused src/messages/#MOSDOp.h# file

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7345 from x11562/rbd_snap_rename

rbd: add "--dest-snap" optional for snap rename

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7393 from tchaikov/wip-cmake

fix the autotools and cmake build (the new fusestore needs libfuse)

Reviewed-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #7403 from rzarzynski/wip-rgw-dumpconlen-cosmetics

rgw: cosmetic improvements in dump_content_length()

rgw: cosmetic improvements in dump_content_length().

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rados: make rados bench metadata backward compatible

We recently made a distinction between op size and object size, and broke
our ability to read metadata from old runs.  Change our format to be
backward compatible.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: disable fuse support if !HAVE_LIBFUSE

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: disable FuseStore if !HAVE_LIBFUSE

Signed-off-by: Kefu Chai <kchai@redhat.com>

make: disable FuseStore if !WITH_FUSE

* configure.ac: "#define HAVE_LIBFUSE" if libfuse is found
* os/Makefile.am: do not compile FuseStore.cc if libfuse is not found

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link against libfuse if HAVE_LIBFUSE

* use ${FUSE_LIBRARIES} instead of "fuse" when referencing libfuse.
* do not compile FuseStore.cc if !HAVE_LIBFUSE.
* CMakeLists.txt: s/HAVE_FUSE/HAVE_LIBFUSE/, to match with the one
  defined for config.h.in.cmake, otherwise we need two vars:
  HAVE_LIBFUSE, and HAVE_FUSE

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove the copyright of FindSnappy.cmake

as we've basically removed all the HyperTable code.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: compact full epochs also

by compacting the ${prefix}.${start}..${prefix}..${end} does not
necessary compact the range of ${prefix}."full_"${start}..
${prefix}."full_"${end}. so when more and more epochs get trimmed
with out a full range compaction, the size of monitor store could
be very large.

Fixes: #14537
Signed-off-by: Kefu Chai <kchai@redhat.com>

Script and Guidelines for mirroring Ceph

This commit introduces a script which admins can use to mirror
the Ceph packages to their local systems.

With this script they can easily sync from a mirror local to them.

The README explains user on how and when to sync the sources and
how they can become a official mirror for Ceph.

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #6875 from dx9/wip-cmake-ccache

cmake: support ccache via a WITH_CCACHE build option

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd: simplify snap rename arguments

Signed-off-by: Na Xie <xie.na@h3c.com>

Merge pull request #7337 from clever215/rbd_nbd

rbd-nbd: Add an argument validation for nbds_max

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7385 from trociny/fix-journal-flush-commit-position

journal: flush commit position on metadata shutdown

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7382 from dillaman/wip-14510

librbd: block maintenance ops until after journal is ready

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

OSD: Deleting transaction object right after applying transaction

Presently, the transaction object is been deleted by the Finisher
thread asynchronously. In heavy load scenario specially if we
unleash journal throttle more, we are seeing high memory usage by
the OSDs because of this. In this new scheme, with the help of
move semantics transaction objects will be deleted synchronously
from the filestore worker threads. We are seeing very much
controllable memory growth now as well as ~3 to 4% cpu usage
benefit because of some reduction of 'new' , 'delete' calls.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

Merge pull request #7259 from trociny/wip-14264

librbd: API: async open and close

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7302 from dillaman/wip-14434

librbd: journal shut down flush race condition

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7299 from dillaman/wip-librbd-update-features

librbd: fix internal handling of dynamic feature updates

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

gitignore: ignore backup files from editors etc.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

remove unused source file

This file was possibly accidentally committed in the tree, thanks
@BranchPredictor for bringing this up in irc

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #7381 from ceph/wip-rgw-static-website-yehuda

RGW static website feature
 - currently disabled by default
 - partial error page handling
 - a few known failures

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Robin H. Johnson <robin.johnson@dreamhost.com>

journal: flush commit position on metadata shutdown

A crash was observed on JournalTrimmer destroy, which was called
by the Journaler after metadata shutdown. JournalTrimmer destructor
also calls flush_commit_position, but at that time metadata m_timer
is already null and it crushed there.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #7223 from liewegas/wip-collection-handles

osd: collection handles

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #7134 from liewegas/wip-fusestore

osd: implement fuse interface for ObjectStore

Reviewed-by: Samuel Just <sjust@redhat.com>

osd: use collection handle where possible

Signed-off-by: Sage Weil <sage@redhat.com>

osd: associate a CollectionHandle& with PGBackend

Make this a ref to the PG's handle because the backend is
constructed before the handle is opened.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: associate a CollectionHandle with each PG

Signed-off-by: Sage Weil <sage@redhat.com>

os/memstore: implement handle-based read methods

Signed-off-by: Sage Weil <sage@redhat.com>

os/memstore: implement open_collection

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: implement CollectionHandle based read methods

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: implement open_collection

Fix up create and remove so that they poison a Collection that
gets removed.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add CollectionHandle wrappers for all read ops

The backend can optionally implement the handle-based variant. If
it does not, the wrappers will call the legacy coll_t version.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add CollectionHandle

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: block maintenance until after journal is ready

Ops were already blocked when the requests were received via
watch/notify.  Requests are now blocked for local requests as
well.

Fixes: #14510
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge branch 'wip-rgw-static-website-yehuda'

Conflicts:
src/rgw/rgw_common.h
src/rgw/rgw_op.h
src/rgw/rgw_rest_s3.cc
src/rgw/rgw_rest_s3.h

os/FuseStore: improve readdir for omap and attr

Use new-style fuse readdir interface for omap and attr.  We aren't
particularly efficient, but we won't completely break if there are a
lot of attrs or omap entries.

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: wait for commit and flush in all updates

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: use offset for readdir

This allows us to list large directories.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/objectstore/test_fuse.sh: simple workunit test

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: add 'type' file in root dir

This also acts as a sentinel to indicate whether we have mounted.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add get_type()

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: drop filestore-specific sanity check

Just let mount() fail

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: add --op mkfs

Signed-off-by: Sage Weil <sage@redhat.com>