LifeCycle feature
As same as amazon S3 interface,"PUT Bucket lifecycle" and
"DELETE Bucket lifecycle" have been implemented,
"GET Bucket lifecycle" not realized yet as S3cmd has not
realize it also.
The feature`s main point is to remove expire file per day.
Files transfer from hot layer to cold layer is not supported.
ToDo:Maybe to transfer from replicate pool to EC pool or
from ssd to sata pool will be valuable.

Now put all buckets which should do lifecycle into shard
objects in .rgw.lc pool.

lifecycle config file format:
<LifecycleConfiguration>
    <Rule>
        <ID>sample-rule</ID>
        <Prefix></Prefix>
        <Status>enable</Status>
        <Expiration>
           <Days>1</Days>
        </Expiration>
    </Rule>
</LifecycleConfiguration>

Signed-off-by: Ji Chen <insomnia@139.com>

deb,rpm: package buffer_fwd.h

Introduced by 206b8e2cc6157451d5c7548e294461408e0b0c8f.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6803 from dachary/wip-13975-systemd-flock

build/ops: systemd ceph-disk unit must not assume /bin/flock

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #6792 from smithfarm/wip-index-20151202

Wip index 20151202

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6795 from ceph/pybind-rados-exec

pybind: support ioctx:exec

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev/index.rst: wholesale refactor

Signed-off-by: Nathan Cutler <ncutler@suse.com>

pybind: support ioctx:exec

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

common/Makefile: ship common/event_socket.h

Broken in commit 5eb64dd324a0e0ec609acf2b178c02dee901aa5d.

Signed-off-by: Sage Weil <sage@redhat.com>

build/ops: systemd ceph-disk unit must not assume /bin/flock

The flock command may be installed elsewhere, depending on the
system. Let the PATH search figure that out.

http://tracker.ceph.com/issues/13975 Fixes: #13975

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #6788 from dachary/wip-13878-ec-pool

test/librados/test.cc: clean up EC pools' crush rules too

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #6665 from trociny/wip-librbd-journal

rbd: journal: configuration via conf, cli, api and some fixes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6800 from dillaman/wip-13969

librbd: fix known test case race condition failures

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: wait for mocked requests to complete

Fixes: #13969
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6641 from aclamk/wip-log-alloc-predictor

common: log: predict log message buffer allocation size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6642 from 0003ydlom/decode_reorder

osd: move newest decode version of MOSDOp and MOSDOpReply to the front

Reviewed-by: Sage Weil <sage@redhat.com>

librbd: do not complete AIO callbacks within caller's thread context

Avoid rare, racy issues when individual requests associated with an AIO
completion can complete prior to marking the completion as ready-to-fire.
Pre-calculate the expected number of individual requests to avoid the
potential re-entrant callback.

Fixes: #13969
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6696 from Ved-vampir/wip_common_plugin

common: add generic plugin infrastructure

Reviewed-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #6779 from ceph/wip-librados-xattr

librados: avoid malloc(0) (which can return NULL on some platforms)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6759 from liewegas/wip-bdev-probe

osd: make backend and block device code a bit more generic

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #6686 from stiopaa1/namespace_buffer

common: use namespace instead of subclasses for buffer

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6535 from mslovy/wip-crc-cache-tracker

common: buffer: add cached_crc and cached_crc_adjust counts to perf dump

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6058 from guangyy/wip-pg-query

osd: dump number of missing objects for each peer with pg query

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5953 from guangyy/wip-13121

osd: support pool level recovery_priority and recovery_op_priority

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #5670 from david-z/master

osd: FileJournal: reduce locking scope in write_aio_bl

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #6798 from trociny/fix-rbd_mirror-bool

librbd: include missing header for bool type

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: include missing header for bool type

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd: alow to specify options for created journal

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: journal options on image create/clone/copy

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: output operator for image options

It is going to be used for diagnostic.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: journal: mark entry committed after replay

After replying a journal entry we have to call committed(). Otherwise,
the entries remain with flag committed=false in JournalMetadata
m_pending_commit_tids forever, which prevents commit position update.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd: add new journal rbd commands

Also, extend 'rbd info' to output journal spec if journaling is enabled.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: humanize ImageWatcher and Journal states in debug log

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: add image pool ID to journal object names.

The data objects are named "journal_data.<pool id>.<image id>.<num>".
This is to avoid conflict when a journal objects pool is used for
several image pools.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: allow alternate pool for journal objects

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: debug: log journal metadata

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: new config options to tweak journal settings

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #6129 from dillaman/wip-13292

RBD pool mirroring configuration API + CLI

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test/librados/test.cc: clean up EC pools' crush rules too

SetUp was adding an erasure-coded pool, which automatically adds
a new crush rule named after the pool, but only removing the
pool.  Remove the crush rule as well.

http://tracker.ceph.com/issues/13878 Fixes: #13878

Signed-off-by: Dan Mick <dan.mick@redhat.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #6780 from smithfarm/wip-index-20151201

doc/dev/index.rst: fix headings

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6786 from oritwas/wip-cmake-fix-radosgw

cmake: add rgw_basic_types.cc to librgw.a

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

doc/dev/index.rst: fix links

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/dev/index.rst: Flesh out the IRC section

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/dev/index.rst: Make Issue tracker a separate chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6789 from dillaman/wip-13969

librbd: partial revert of commit 9b0e359

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6748 from liewegas/wip-fix-cephtool

qa/workunits/cephtool/test.sh: no ./

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5941 from jcsp/wip-12137

#12137: cephfs-data-scan scan_frags

Merge pull request #6777 from dillaman/wip-librbd-lockdep

librbd: journaling-related lock dependency cleanup

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6762 from dillaman/wip-13924

journal: fire replay complete event after reading last object

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6703 from dillaman/wip-tests-librbd-gmock-state-machines

librbd: initial collection of state machine unit tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: partial revert of commit 9b0e359

Fixes: #13969
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc/dev/index.rst: start writing Bugfixing chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6690 from branch-predictor/bp-smaller-pglog-2

osd: slightly reduce actual size of pg_log_entry_t

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6689 from 0003ydlom/doubled_cond

osd: avoid duplicate op->mark_started in ReplicatedBackend

Reviewed-by: xinxin shu <xinxin.shu@intel.com>

Merge pull request #6681 from ifed01/wip-objstore-bench-stabilization

ceph_objectstore_bench: fix race condition, bugs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6677 from jack-changtao/test-bench

rados: bench: fix off-by-one to avoid writing past object_size

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6555 from mslovy/wip-reduce-omap-transaction

osd: don't update epoch and rollback_info objects attrs if there is no need

Reviewed-by: Sage Weil <sage@redhat.com>

cmake: add rgw_basic_types.cc to librgw.a

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

doc/dev/index.rst: rewrite introductory material

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6784 from ceph/wip-5073

rgw multitenancy

Buckets and users can now have a tenant associated with them. Buckets can be specified as [<tenant>:]<bucket> and users can be specified as [<tenant>$]<uid>. The existing users and buckets are part of the default global tenant.
This is first phase. Future work will add tighter integration (e.g., swift tenant mapping, s3 subdomain tenant mapping, etc.)

Reviewed-by: Pete Zaitcev <zaitcev@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

qa: add a test for the recovery_priority/recovery_op_priority pool settings.

Signed-off-by: Guang Yang <yguang@yahoo-inc.com>

mon: support recovery_priority and recovery_op_priority with monitor commands

Fixes: 13121
Signed-off-by: Guang Yang yguang@yahoo-inc.com

osd: prioritize recovery based on pool's customized priority

Fixes: 13121
Signed-off-by: Guang Yang <yguang@yahoo-inc.com>

doc/dev/index.rst: fix headings

This commit fixes the headings so they render properly on http://ceph.com

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6727 from smithfarm/wip-index

doc/dev/index.rst: begin writing Contributing to Ceph

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/dev/index.rst: begin writing Contributing to Ceph

Replace existing index.rst with the beginnings of a comprehensive
developer guide.

Move the old index.rst to index-old.rst.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

common: lockdep now tracks lock ordering when backtraces disabled

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: correct lock ordering issues discovered by lockdep

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6775 from mkoderer/doc/fixowner

doc: remove unnecessary period in headline

Reviewed-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

doc: remove unnecessary period in headline

Signed-off-by: Marc Koderer <marc@koderer.com>

journal: correct lock ordering issues discovered by lockdep

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: remove comments

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a minor cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't re-set bucket tenant and name when selecting location

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix a typo

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't allow cross-tenant bucket creation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: avoid calling rgw_make_bucket_entry_name() when not needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: objexp hint name backward compatibility

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: inherit bucket tenant from user if not specified

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6739 from dreamhost/wip-13920

Gentoo: _FORTIFY_SOURCE fix.

Reviewed-by: Samuel Just <sjust@redhat.com>

tests: added new rbd mirroring CLI commands

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6771 from dillaman/wip-backtrace-fixes

common: re-enable backtrace support

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rbd: add new 'mirror pool' rbd commands

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: librbd new pool mirror API methods

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new pool mirror API methods

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph-dencoder: new cls::rbd::MirrorPeer type

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: new cls_rbd methods for pool mirroring

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: new methods for handling mirroring

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: initial set of test cases for op state machines

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: snap create doesn't properly handle race conditions

The ESTALE result code was being filtered out, preventing the retry
logic.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: use md_ctx for self-managed snapshot management

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: convert op state machines to templates

Using templates allows the creation of mocked test cases to
verify non-librados expectations.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: add gmock unit tests for librbd object map state machines

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ensure object map is invalidated on disk if forced

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6657 from kylinstorage/wip-rbd-nbd

rbd-nbd: network block device (NBD) support for RBD

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6769 from aiicore/configure_fix

configure.ac: macro fix

Reviewed-by: Boris Ranto <branto@redhat.com>

common: PluginRegistry modification

common: PluginRegistry

Generic plugin loader.  Based on the ErasureCodePlugin.

Signed-off-by: Sage Weil <sage@redhat.com>
Conflicts:
src/common/Makefile.am
src/common/ceph_context.cc
src/common/ceph_context.h

missing files

configure.ac: macro fix

Configure fails with autoconf 2.63 on Centos 6.6 with:

./configure: line 34026: syntax error near unexpected token `newline'
./configure: line 34026: `  yes:no:'

Signed-off-by: Igor Podoski <igor.podoski@ts.fujitsu.com>

Merge pull request #5685 from ceph/wip-12133

cephfs-data-scan: scrub tag filtering (#12133 and #12145)

tools/cephfs: enable tag filter in DataScan

Fixes: #12133
Signed-off-by: John Spray <john.spray@redhat.com>