v0.94 Hammer (draft)
============
+This major release is expected to form the basis of the next long-term
+stable series. It is intended to supercede v0.80.x Firefly.
+
+Highlights since Giant include:
+
+* *RADOS Performance*: a range of improvements have been made in the
+ OSD and client-side librados code that improve the throughput on
+ flash backends and improve parallelism and scaling on fast machines.
+* *RGW object versioning*: RGW now supports the S3 object versioning
+ API, which preserves old version of objects instead of overwriting
+ them.
+* *RGW bucket sharding*: RGW can now shard the bucket index for large
+ buckets across, improving performance for very large buckets.
+* *RBD object maps*: RBD now has an object map function that tracks
+ which parts of the image are allocating, improving performance for
+ clones and for commands like export and delete.
+* *RBD mandatory locking*: RBD has a new mandatory locking framework
+ (still disabled by default) that adds additional safeguards to
+ prevent multiple clients from using the same image at the same time.
+* *RBD copy-on-read*: RBD now supports copy-on-read for image clones,
+ improving performance for some workloads.
+* *CephFS snapshot improvements*: Many many bugs have been fixed with
+ CephFS snapshots. Although they are still disabled by default,
+ stability has improved significantly.
+* *CephFS Recovery tools*: We have built some journal recovery and
+ diagnostic tools. Stability and performance of single-MDS systems is
+ vastly improved in Giant, and more improvements have been made now
+ in Hammer. Although we still recommend caution when storing
+ important data in CephFS, we do encourage testing for non-critical
+ workloads so that we can better guage the feature, usability,
+ performance, and stability gaps.
+* *CRUSH improvements*: We have added a new straw2 bucket algorithm
+ that reduces the amount of data migration required when changes are
+ made to the cluster.
+* *RADOS cache tiering*: A series of changes have been made in the
+ cache tiering code that improve performance and reduce latency.
+* *RDMA support*: There is now experimental support the RDMA via the
+ Accelio (libxio) library.
+* *New administrator commands*: The 'ceph osd df' command shows
+ pertinent details on OSD disk utilizations. The 'ceph pg ls ...'
+ command makes it much simpler to query PG states while diagnosing
+ cluster issues.
+
+Other highlights since Firefly include:
+
+* *CephFS*: we have fixed a raft of bugs in CephFS and built some
+ basic journal recovery and diagnostic tools. Stability and
+ performance of single-MDS systems is vastly improved in Giant.
+ Although we do not yet recommend CephFS for production deployments,
+ we do encourage testing for non-critical workloads so that we can
+ better guage the feature, usability, performance, and stability
+ gaps.
+* *Local Recovery Codes*: the OSDs now support an erasure-coding scheme
+ that stores some additional data blocks to reduce the IO required to
+ recover from single OSD failures.
+* *Degraded vs misplaced*: the Ceph health reports from 'ceph -s' and
+ related commands now make a distinction between data that is
+ degraded (there are fewer than the desired number of copies) and
+ data that is misplaced (stored in the wrong location in the
+ cluster). The distinction is important because the latter does not
+ compromise data safety.
+* *Tiering improvements*: we have made several improvements to the
+ cache tiering implementation that improve performance. Most
+ notably, objects are not promoted into the cache tier by a single
+ read; they must be found to be sufficiently hot before that happens.
+* *Monitor performance*: the monitors now perform writes to the local
+ data store asynchronously, improving overall responsiveness.
+* *Recovery tools*: the ceph-objectstore-tool is greatly expanded to
+ allow manipulation of an individual OSDs data store for debugging
+ and repair purposes. This is most heavily used by our QA
+ infrastructure to exercise recovery code.
+
+I would like to take this opportunity to call out the amazing growth
+in contributors to Ceph beyond the core development team from Inktank.
+Hammer features major new features and improvements from Intel,
+UnitedStack, Yahoo, UbuntuKylin, CohortFS, Mellanox, CERN, Deutsche
+Telekom, Mirantis, and SanDisk.
+
+Dedication
+----------
+
+This release is dedicated in memoriam to Sandon Van Ness, aka
+Houkouonchi, a valued member of the Ceph team who unexpectedly passed
+away a few weeks ago. Sandon was responsible for maintaining the
+large and complex Sepia lab that houses the Ceph project's build and
+test infrastructure. His efforts have made an important impact on our
+ability to reliably test Ceph with a relatively small group of people.
+He was a valued member of the team and we will miss him.
Upgrading
---------
+* If your existing cluster is running a version older than v0.80.x
+ Firefly, please first upgrade to the latest Firefly release before
+ moving on to Giant. We have not tested upgrades directly from
+ Emperor, Dumpling, or older releases.
+
+ We *have* tested:
+
+ * Firefly to Hammer
+ * Firefly to Giant to Hammer
+ * Dumpling to Firefly to Hammer
+
+* Please upgrade daemons in the following order:
+
+ #. Monitors
+ #. OSDs
+ #. MDSs and/or radosgw
+
+ Note that the relative ordering of OSDs and monitors should not matter, but
+ we primarily tested upgrading monitors first.
+
* The ceph-osd daemons will perform a disk-format upgrade improve the
PG metadata layout and to repair a minor bug in the on-disk format.
It may take a minute or two for this to complete, depending on how
many objects are stored on the node; do not be alarmed if they do
not marked "up" by the cluster immediately after starting.
+Upgrading from v0.80.x Giant
+----------------------------
+
* librbd and librados include lttng tracepoints on distros with
liblttng 2.4 or later (only Ubuntu Trusty for the ceph.com
packages). When running a daemon that uses these libraries, i.e. an
to take precedence. This change should only affect users who have
made non-standard changes to their radosgw configuration.
+* If you are upgrading specifically from v0.92, you must stop all OSD
+ daemons and flush their journals (``ceph-osd -i NNN
+ --flush-journal'') before upgrading. There was a transaction
+ encoding bug in v0.92 that broke compatibility. Upgrading from v0.93,
+ v0.91, or anything earlier is safe.
+
+* The experimental 'keyvaluestore-dev' OSD backend has been renamed
+ 'keyvaluestore' (for simplicity) and marked as experimental. To
+ enable this untested feature and acknowledge that you understand
+ that it is untested and may destroy data, you need to add the
+ following to your ceph.conf::
+
+ enable experimental unrecoverable data corrupting featuers = keyvaluestore
+
+* The following librados C API function calls take a 'flags' argument whose value
+ is now correctly interpreted:
+
+ rados_write_op_operate()
+ rados_aio_write_op_operate()
+ rados_read_op_operate()
+ rados_aio_read_op_operate()
+
+ The flags were not correctly being translated from the librados constants to the
+ internal values. Now they are. Any code that is passing flags to these methods
+ should be audited to ensure that they are using the correct LIBRADOS_OP_FLAG_*
+ constants.
+
+* The 'rados' CLI 'copy' and 'cppool' commands now use the copy-from operation,
+ which means the latest CLI cannot run these commands against pre-firefly OSDs.
+
+* The librados watch/notify API now includes a watch_flush() operation to flush
+ the async queue of notify operations. This should be called by any watch/notify
+ user prior to rados_shutdown().
+
+* The 'category' field for objects has been removed. This was originally added
+ to track PG stat summations over different categories of objects for use by
+ radosgw. It is no longer has any known users and is prone to abuse because it
+ can lead to a pg_stat_t structure that is unbounded. The librados API calls
+ that accept this field now ignore it, and the OSD no longers tracks the
+ per-category summations.
+
+* The output for 'rados df' has changed. The 'category' level has been
+ eliminated, so there is now a single stat object per pool. The structure of
+ the JSON output is different, and the plaintext output has one less column.
+
+* The 'rados create <objectname> [category]' optional category argument is no
+ longer supported or recognized.
+
+* rados.py's Rados class no longer has a __del__ method; it was causing
+ problems on interpreter shutdown and use of threads. If your code has
+ Rados objects with limited lifetimes and you're concerned about locked
+ resources, call Rados.shutdown() explicitly.
+
+* There is a new version of the librados watch/notify API with vastly
+ improved semantics. Any applications using this interface are
+ encouraged to migrate to the new API. The old API calls are marked
+ as deprecated and will eventually be removed.
+
+* The librados rados_unwatch() call used to be safe to call on an
+ invalid handle. The new version has undefined behavior when passed
+ a bogus value (for example, when rados_watch() returns an error and
+ handle is not defined).
+
+* The structure of the formatted 'pg stat' command is changed for the
+ portion that counts states by name to avoid using the '+' character
+ (which appears in state names) as part of the XML token (it is not
+ legal).
+
+* Previously, the formatted output of 'ceph pg stat -f ...' was a full
+ pg dump that included all metadata about all PGs in the system. It
+ is now a concise summary of high-level PG stats, just like the
+ unformatted 'ceph pg stat' command.
+
+* All JSON dumps of floating point values were incorrecting surrounding the
+ value with quotes. These quotes have been removed. Any consumer of structured
+ JSON output that was consuming the floating point values was previously having
+ to interpret the quoted string and will most likely need to be fixed to take
+ the unquoted number.
+
+* New ability to list all objects from all namespaces can fail or
+ return incomplete results when not all OSDs have been upgraded.
+ Features rados --all ls, rados cppool, rados export, rados
+ cache-flush-evict-all and rados cache-try-flush-evict-all can also
+ fail or return incomplete results.
+
+* Due to a change in the Linux kernel version 3.18 and the limits of the FUSE
+ interface, ceph-fuse needs be mounted as root on at least some systems. See
+ issues #9997, #10277, and #10542 for details.
+
+Upgrading from v0.80x Firefly (additional notes)
+------------------------------------------------
+
+* The client-side caching for librbd is now enabled by default (rbd
+ cache = true). A safety option (rbd cache writethrough until flush
+ = true) is also enabled so that writeback caching is not used until
+ the library observes a 'flush' command, indicating that the librbd
+ users is passing that operation through from the guest VM. This
+ avoids potential data loss when used with older versions of qemu
+ that do not support flush.
+
+ leveldb_write_buffer_size = 8*1024*1024 = 33554432 // 8MB
+ leveldb_cache_size = 512*1024*1204 = 536870912 // 512MB
+ leveldb_block_size = 64*1024 = 65536 // 64KB
+ leveldb_compression = false
+ leveldb_log = ""
+
+ OSDs will still maintain the following osd-specific defaults:
+
+ leveldb_log = ""
+
+* The 'rados getxattr ...' command used to add a gratuitous newline to the attr
+ value; it now does not.
+
+* The ``*_kb perf`` counters on the monitor have been removed. These are
+ replaced with a new set of ``*_bytes`` counters (e.g., ``cluster_osd_kb`` is
+ replaced by ``cluster_osd_bytes``).
+
+* The ``rd_kb`` and ``wr_kb`` fields in the JSON dumps for pool stats (accessed
+ via the ``ceph df detail -f json-pretty`` and related commands) have been
+ replaced with corresponding ``*_bytes`` fields. Similarly, the
+ ``total_space``, ``total_used``, and ``total_avail`` fields are replaced with
+ ``total_bytes``, ``total_used_bytes``, and ``total_avail_bytes`` fields.
+
+* The ``rados df --format=json`` output ``read_bytes`` and ``write_bytes``
+ fields were incorrectly reporting ops; this is now fixed.
+
+* The ``rados df --format=json`` output previously included ``read_kb`` and
+ ``write_kb`` fields; these have been removed. Please use ``read_bytes`` and
+ ``write_bytes`` instead (and divide by 1024 if appropriate).
+
+* The experimental keyvaluestore-dev OSD backend had an on-disk format
+ change that prevents existing OSD data from being upgraded. This
+ affects developers and testers only.
+
+* mon-specific and osd-specific leveldb options have been removed.
+ From this point onward users should use the `leveldb_*` generic
+ options and add the options in the appropriate sections of their
+ configuration files. Monitors will still maintain the following
+ monitor-specific defaults:
+
+ leveldb_write_buffer_size = 8*1024*1024 = 33554432 // 8MB
+ leveldb_cache_size = 512*1024*1204 = 536870912 // 512MB
+ leveldb_block_size = 64*1024 = 65536 // 64KB
+ leveldb_compression = false
+ leveldb_log = ""
+
+ OSDs will still maintain the following osd-specific defaults:
+
+ leveldb_log = ""
+
+* CephFS support for the legacy anchor table has finally been removed.
+ Users with file systems created before firefly should ensure that inodes
+ with multiple hard links are modified *prior* to the upgrade to ensure that
+ the backtraces are written properly. For example::
+
+ sudo find /mnt/cephfs -type f -links +1 -exec touch \{\} \;
+
+* We disallow nonsensical 'tier cache-mode' transitions. From this point
+ onward, 'writeback' can only transition to 'forward' and 'forward'
+ can transition to 1) 'writeback' if there are dirty objects, or 2) any if
+ there are no dirty objects.
+
+
+Notable changes since v0.93
+---------------------------
+
+* build: a few cmake fixes (Matt Benjamin)
+* build: fix build on RHEL/CentOS 5.9 (Rohan Mars)
+* build: reorganize Makefile to allow modular builds (Boris Ranto)
+* ceph-fuse: be more forgiving on remount (#10982 Greg Farnum)
+* ceph: improve CLI parsing (#11093 David Zafman)
+* crush: fix parsing of straw2 buckets (#11015 Sage Weil)
+* doc: update man pages (David Zafman)
+* librados: fix leak in C_TwoContexts (Xiong Yiliang)
+* librados: fix leak in watch/notify path (Sage Weil)
+* librbd: fix and improve AIO cache invalidation (#10958 Jason Dillaman)
+* librbd: fix memory leak (Jason Dillaman)
+* librbd: fix ordering/queueing of resize operations (Jason Dillaman)
+* librbd: validate image is r/w on resize/flatten (Jason Dillaman)
+* librbd: various internal locking fixes (Jason Dillaman)
+* mon: add bootstrap-rgw profile (Sage Weil)
+* mon: do not pollute mon dir with CSV files from CRUSH check (Loic Dachary)
+* mon: fix clock drift time check interval (#10546 Joao Eduardo Luis)
+* mon: fix units in store stats (Joao Eduardo Luis)
+* mon: improve error handling on erasure code profile set (#10488, #11144 Loic Dachary)
+* mon: set {read,write}_tier on 'osd tier add-cache ...' (Jianpeng Ma)
+* ms: xio: fix misc bugs (Matt Benjamin, Vu Pham)
+* osd: DBObjectMap: fix locking to prevent rare crash (#9891 Samuel Just)
+* osd: fix and document last_epoch_started semantics (Samuel Just)
+* osd: fix divergent entry handling on PG split (Samuel Just)
+* osd: fix leak on shutdown (Kefu Chai)
+* osd: fix recording of digest on scrub (Samuel Just)
+* osd: fix whiteout handling (Sage Weil)
+* rbd: allow v2 striping parameters for clones and imports (Jason Dillaman)
+* rbd: fix formatted output of image features (Jason Dillaman)
+* rbd: updat eman page (Ilya Dryomov)
+* rgw: don't overwrite bucket/object owner when setting ACLs (#10978 Yehuda Sadeh)
+* rgw: enable IPv6 for civetweb (#10965 Yehuda Sadeh)
+* rgw: fix sysvinit script when rgw_socket_path is not defined (#11159 Yehuda Sadeh, Dan Mick)
+* rgw: pass civetweb configurables through (#10907 Yehuda Sadeh)
+* rgw: use new watch/notify API (Yehuda Sadeh, Sage Weil)
+
+Notable changes since v0.87.x Giant
+-----------------------------------
+
+* add experimental features option (Sage Weil)
+* arch: fix NEON feaeture detection (#10185 Loic Dachary)
+* asyncmsgr: misc fixes (Haomai Wang)
+* buffer: add 'shareable' construct (Matt Benjamin)
+* buffer: add list::get_contiguous (Sage Weil)
+* buffer: avoid rebuild if buffer already contiguous (Jianpeng Ma)
+* build: CMake support (Ali Maredia, Casey Bodley, Adam Emerson, Marcus Watts, Matt Benjamin)
+* build: a few cmake fixes (Matt Benjamin)
+* build: aarch64 build fixes (Noah Watkins, Haomai Wang)
+* build: adjust build deps for yasm, virtualenv (Jianpeng Ma)
+* build: fix 'make check' races (#10384 Loic Dachary)
+* build: fix build on RHEL/CentOS 5.9 (Rohan Mars)
+* build: fix pkg names when libkeyutils is missing (Pankag Garg, Ken Dreyer)
+* build: improve build dependency tooling (Loic Dachary)
+* build: reorganize Makefile to allow modular builds (Boris Ranto)
+* build: support for jemalloc (Shishir Gowda)
+* ceph-disk: Scientific Linux support (Dan van der Ster)
+* ceph-disk: allow journal partition re-use (#10146 Loic Dachary, Dav van der Ster)
+* ceph-disk: call partx/partprobe consistency (#9721 Loic Dachary)
+* ceph-disk: do not re-use partition if encryption is required (Loic Dachary)
+* ceph-disk: fix dmcrypt key permissions (Loic Dachary)
+* ceph-disk: fix umount race condition (#10096 Blaine Gardner)
+* ceph-disk: improved systemd support (Owen Synge)
+* ceph-disk: init=none option (Loic Dachary)
+* ceph-disk: misc fixes (Christos Stavrakakis)
+* ceph-disk: respect --statedir for keyring (Loic Dachary)
+* ceph-disk: set guid if reusing journal partition (Dan van der Ster)
+* ceph-disk: support LUKS for encrypted partitions (Andrew Bartlett, Loic Dachary)
+* ceph-fuse, libcephfs: POSIX file lock support (Yan, Zheng)
+* ceph-fuse, libcephfs: allow xattr caps in inject_release_failure (#9800 John Spray)
+* ceph-fuse, libcephfs: fix I_COMPLETE_ORDERED checks (#9894 Yan, Zheng)
+* ceph-fuse, libcephfs: fix cap flush overflow (Greg Farnum, Yan, Zheng)
+* ceph-fuse, libcephfs: fix root inode xattrs (Yan, Zheng)
+* ceph-fuse, libcephfs: preserve dir ordering (#9178 Yan, Zheng)
+* ceph-fuse, libcephfs: trim inodes before reconnecting to MDS (Yan, Zheng)
+* ceph-fuse,libcephfs: add support for O_NOFOLLOW and O_PATH (Greg Farnum)
+* ceph-fuse,libcephfs: resend requests before completing cap reconnect (#10912 Yan, Zheng)
+* ceph-fuse: be more forgiving on remount (#10982 Greg Farnum)
+* ceph-fuse: fix dentry invalidation on 3.18+ kernels (#9997 Yan, Zheng)
+* ceph-fuse: fix kernel cache trimming (#10277 Yan, Zheng)
+* ceph-fuse: select kernel cache invalidation mechanism based on kernel version (Greg Farnum)
+* ceph-monstore-tool: fix shutdown (#10093 Loic Dachary)
+* ceph-monstore-tool: fix/improve CLI (Joao Eduardo Luis)
+* ceph-objectstore-tool: fix import (#10090 David Zafman)
+* ceph-objectstore-tool: improved import (David Zafman)
+* ceph-objectstore-tool: many improvements and tests (David Zafman)
+* ceph-objectstore-tool: many many improvements (David Zafman)
+* ceph-objectstore-tool: misc improvements, fixes (#9870 #9871 David Zafman)
+* ceph.spec: package rbd-replay-prep (Ken Dreyer)
+* ceph: add 'ceph osd df [tree]' command (#10452 Mykola Golub)
+* ceph: do not parse injectargs twice (Loic Dachary)
+* ceph: fix 'ceph tell ...' command validation (#10439 Joao Eduardo Luis)
+* ceph: improve 'ceph osd tree' output (Mykola Golub)
+* ceph: improve CLI parsing (#11093 David Zafman)
+* ceph: make 'ceph -s' output more readable (Sage Weil)
+* ceph: make 'ceph -s' show PG state counts in sorted order (Sage Weil)
+* ceph: make 'ceph tell mon.* version' work (Mykola Golub)
+* ceph: new 'ceph tell mds.$name_or_rank_or_gid' (John Spray)
+* ceph: show primary-affinity in 'ceph osd tree' (Mykola Golub)
+* ceph: test robustness (Joao Eduardo Luis)
+* ceph_objectstore_tool: behave with sharded flag (#9661 David Zafman)
+* cephfs-journal-tool: add recover_dentries function (#9883 John Spray)
+* cephfs-journal-tool: fix journal import (#10025 John Spray)
+* cephfs-journal-tool: skip up to expire_pos (#9977 John Spray)
+* cleanup rados.h definitions with macros (Ilya Dryomov)
+* common: add 'perf reset ...' admin command (Jianpeng Ma)
+* common: add TableFormatter (Andreas Peters)
+* common: add newline to flushed json output (Sage Weil)
+* common: check syncfs() return code (Jianpeng Ma)
+* common: do not unlock rwlock on destruction (Federico Simoncelli)
+* common: filtering for 'perf dump' (John Spray)
+* common: fix Formatter factory breakage (#10547 Loic Dachary)
+* common: fix block device discard check (#10296 Sage Weil)
+* common: make json-pretty output prettier (Sage Weil)
+* common: remove broken CEPH_LOCKDEP optoin (Kefu Chai)
+* common: shared_cache unit tests (Cheng Cheng)
+* common: support new gperftools header locations (Key Dreyer)
+* config: add $cctid meta variable (Adam Crume)
+* crush: fix buffer overrun for poorly formed rules (#9492 Johnu George)
+* crush: fix detach_bucket (#10095 Sage Weil)
+* crush: fix parsing of straw2 buckets (#11015 Sage Weil)
+* crush: fix several bugs in adjust_item_weight (Rongze Zhu)
+* crush: fix tree bucket behavior (Rongze Zhu)
+* crush: improve constness (Loic Dachary)
+* crush: new and improved straw2 bucket type (Sage Weil, Christina Anderson, Xiaoxi Chen)
+* crush: straw bucket weight calculation fixes (#9998 Sage Weil)
+* crush: update tries stats for indep rules (#10349 Loic Dachary)
+* crush: use larger choose_tries value for erasure code rulesets (#10353 Loic Dachary)
+* crushtool: add --location <id> command (Sage Weil, Loic Dachary)
+* debian,rpm: move RBD udev rules to ceph-common (#10864 Ken Dreyer)
+* debian: split python-ceph into python-{rbd,rados,cephfs} (Boris Ranto)
+* default to libnss instead of crypto++ (Federico Gimenez)
+* doc: CephFS disaster recovery guidance (John Spray)
+* doc: CephFS for early adopters (John Spray)
+* doc: add build-doc guidlines for Fedora and CentOS/RHEL (Nilamdyuti Goswami)
+* doc: add dumpling to firefly upgrade section (#7679 John Wilkins)
+* doc: ceph osd reweight vs crush weight (Laurent Guerby)
+* doc: do not suggest dangerous XFS nobarrier option (Dan van der Ster)
+* doc: document erasure coded pool operations (#9970 Loic Dachary)
+* doc: document the LRC per-layer plugin configuration (Yuan Zhou)
+* doc: enable rbd cache on openstack deployments (Sebastien Han)
+* doc: erasure code doc updates (Loic Dachary)
+* doc: file system osd config settings (Kevin Dalley)
+* doc: fix OpenStack Glance docs (#10478 Sebastien Han)
+* doc: improved installation nots on CentOS/RHEL installs (John Wilkins)
+* doc: key/value store config reference (John Wilkins)
+* doc: misc cleanups (Adam Spiers, Sebastien Han, Nilamdyuti Goswami, Ken Dreyer, John Wilkins)
+* doc: misc improvements (Nilamdyuti Goswami, John Wilkins, Chris Holcombe)
+* doc: misc updates (#9793 #9922 #10204 #10203 Travis Rhoden, Hazem, Ayari, Florian Coste, Andy Allan, Frank Yu, Baptiste Veuillez-Mainard, Yuan Zhou, Armando Segnini, Robert Jansen, Tyler Brekke, Viktor Suprun)
+* doc: misc updates (Alfredo Deza, VRan Liu)
+* doc: misc updates (Nilamdyuti Goswami, John Wilkins)
+* doc: new man pages (Nilamdyuti Goswami)
+* doc: preflight doc fixes (John Wilkins)
+* doc: replace cloudfiles with swiftclient Python Swift example (Tim Freund)
+* doc: update PG count guide (Gerben Meijer, Laurent Guerby, Loic Dachary)
+* doc: update man pages (David Zafman)
+* doc: update openstack docs for Juno (Sebastien Han)
+* doc: update release descriptions (Ken Dreyer)
+* doc: update sepia hardware inventory (Sandon Van Ness)
+* erasure-code: add mSHEC erasure code support (Takeshi Miyamae)
+* erasure-code: improved docs (#10340 Loic Dachary)
+* erasure-code: set max_size to 20 (#10363 Loic Dachary)
+* fix cluster logging from non-mon daemons (Sage Weil)
+* init-ceph: check for systemd-run before using it (Boris Ranto)
+* install-deps.sh: do not require sudo when root (Loic Dachary)
+* keyvaluestore: misc fixes (Haomai Wang)
+* keyvaluestore: performance improvements (Haomai Wang)
+* libcephfs,ceph-fuse: add 'status' asok (John Spray)
+* libcephfs,ceph-fuse: fix getting zero-length xattr (#10552 Yan, Zheng)
+* libcephfs: fix dirfrag trimming (#10387 Yan, Zheng)
+* libcephfs: fix mount timeout (#10041 Yan, Zheng)
+* libcephfs: fix test (#10415 Yan, Zheng)
+* libcephfs: fix use-afer-free on umount (#10412 Yan, Zheng)
+* libcephfs: include ceph and git version in client metadata (Sage Weil)
+* librados, osd: new watch/notify implementation (Sage Weil)
+* librados: add blacklist_add convenience method (Jason Dillaman)
+* librados: add rados_pool_get_base_tier() call (Adam Crume)
+* librados: add watch_flush() operation (Sage Weil, Haomai Wang)
+* librados: avoid memcpy on getxattr, read (Jianpeng Ma)
+* librados: cap buffer length (Loic Dachary)
+* librados: create ioctx by pool id (Jason Dillaman)
+* librados: do notify completion in fast-dispatch (Sage Weil)
+* librados: drop 'category' feature (Sage Weil)
+* librados: expose rados_{read|write}_op_assert_version in C API (Kim Vandry)
+* librados: fix infinite loop with skipped map epochs (#9986 Ding Dinghua)
+* librados: fix iterator operator= bugs (#10082 David Zafman, Yehuda Sadeh)
+* librados: fix leak in C_TwoContexts (Xiong Yiliang)
+* librados: fix leak in watch/notify path (Sage Weil)
+* librados: fix null deref when pool DNE (#9944 Sage Weil)
+* librados: fix objecter races (#9617 Josh Durgin)
+* librados: fix pool deletion handling (#10372 Sage Weil)
+* librados: fix pool name caching (#10458 Radoslaw Zarzynski)
+* librados: fix resource leak, misc bugs (#10425 Radoslaw Zarzynski)
+* librados: fix some watch/notify locking (Jason Dillaman, Josh Durgin)
+* librados: fix timer race from recent refactor (Sage Weil)
+* librados: new fadvise API (Ma Jianpeng)
+* librados: only export public API symbols (Jason Dillaman)
+* librados: remove shadowed variable (Kefu Chain)
+* librados: translate op flags from C APIs (Matthew Richards)
+* libradosstriper: fix remove() (Dongmao Zhang)
+* libradosstriper: fix shutdown hang (Dongmao Zhang)
+* libradosstriper: fix stat strtoll (Dongmao Zhang)
+* libradosstriper: fix trunc method (#10129 Sebastien Ponce)
+* libradosstriper: fix write_full when ENOENT (#10758 Sebastien Ponce)
+* libradosstriper: misc fixes (Sebastien Ponce)
+* librbd: CRC protection for RBD image map (Jason Dillaman)
+* librbd: add missing python docstrings (Jason Dillaman)
+* librbd: add per-image object map for improved performance (Jason Dillaman)
+* librbd: add readahead (Adam Crume)
+* librbd: add support for an "object map" indicating which objects exist (Jason Dillaman)
+* librbd: adjust internal locking (Josh Durgin, Jason Dillaman)
+* librbd: better handling of watch errors (Jason Dillaman)
+* librbd: complete pending ops before closing image (#10299 Josh Durgin)
+* librbd: coordinate maint operations through lock owner (Jason Dillaman)
+* librbd: copy-on-read (Min Chen, Li Wang, Yunchuan Wen, Cheng Cheng, Jason Dillaman)
+* librbd: differentiate between R/O vs R/W features (Jason Dillaman)
+* librbd: don't close a closed parent in failure path (#10030 Jason Dillaman)
+* librbd: enforce write ordering with a snapshot (Jason Dillaman)
+* librbd: exclusive image locking (Jason Dillaman)
+* librbd: fadvise API (Ma Jianpeng)
+* librbd: fadvise-style hints; add misc hints for certain operations (Jianpeng Ma)
+* librbd: fix and improve AIO cache invalidation (#10958 Jason Dillaman)
+* librbd: fix cache tiers in list_children and snap_unprotect (Adam Crume)
+* librbd: fix coverity false-positives (Jason Dillaman)
+* librbd: fix diff test (#10002 Josh Durgin)
+* librbd: fix list_children from invalid pool ioctxs (#10123 Jason Dillaman)
+* librbd: fix locking for readahead (#10045 Jason Dillaman)
+* librbd: fix memory leak (Jason Dillaman)
+* librbd: fix ordering/queueing of resize operations (Jason Dillaman)
+* librbd: fix performance regression in ObjectCacher (#9513 Adam Crume)
+* librbd: fix snap create races (Jason Dillaman)
+* librbd: fix write vs import race (#10590 Jason Dillaman)
+* librbd: flush AIO operations asynchronously (#10714 Jason Dillaman)
+* librbd: gracefully handle deleted/renamed pools (#10270 Jason Dillaman)
+* librbd: lttng tracepoints (Adam Crume)
+* librbd: make async versions of long-running maint operations (Jason Dillaman)
+* librbd: misc fixes (Xinxin Shu, Jason Dillaman)
+* librbd: mock tests (Jason Dillaman)
+* librbd: only export public API symbols (Jason Dillaman)
+* librbd: optionally blacklist clients before breaking locks (#10761 Jason Dillaman)
+* librbd: prevent copyup during shrink (Jason Dillaman)
+* librbd: refactor unit tests to use fixtures (Jason Dillaman)
+* librbd: validate image is r/w on resize/flatten (Jason Dillaman)
+* librbd: various internal locking fixes (Jason Dillaman)
+* many coverity fixes (Danny Al-Gaaf)
+* many many coverity cleanups (Danny Al-Gaaf)
+* mds: 'flush journal' admin command (John Spray)
+* mds: ENOSPC and OSDMap epoch barriers (#7317 John Spray)
+* mds: a whole bunch of initial scrub infrastructure (Greg Farnum)
+* mds: add cephfs-table-tool (John Spray)
+* mds: asok command for fetching subtree map (John Spray)
+* mds: avoid sending traceless replies in most cases (Yan, Zheng)
+* mds: constify MDSCacheObjects (John Spray)
+* mds: dirfrag buf fix (Yan, Zheng)
+* mds: disallow most commands on inactive MDS's (Greg Farnum)
+* mds: drop dentries, leases on de* The ceph-osd daemons will perform a disk-format upgrade improve the
+leted directories (#10164 Yan, Zheng)
+* mds: export dir asok command (John Spray)
+* mds: fix MDLog IO callback deadlock (John Spray)
+* mds: fix compat_version for MClientSession (#9945 John Spray)
+* mds: fix deadlock during journal probe vs purge (#10229 Yan, Zheng)
+* mds: fix race trimming log segments (Yan, Zheng)
+* mds: fix reply snapbl (Yan, Zheng)
+* mds: fix sessionmap lifecycle bugs (Yan, Zheng)
+* mds: fix stray/purge perfcounters (#10388 John Spray)
+* mds: handle heartbeat_reset during shutdown (#10382 John Spray)
+* mds: handle zero-size xattr (#10335 Yan, Zheng)
+* mds: initialize root inode xattr version (Yan, Zheng)
+* mds: introduce auth caps (John Spray)
+* mds: many many snapshot-related fixes (Yan, Zheng)
+* mds: misc bugs (Greg Farnum, John Spray, Yan, Zheng, Henry Change)
+* mds: refactor, improve Session storage (John Spray)
+* mds: store backtrace for stray dir (Yan, Zheng)
+* mds: subtree quota support (Yunchuan Wen)
+* mds: verify backtrace when fetching dirfrag (#9557 Yan, Zheng)
+* memstore: free space tracking (John Spray)
+* misc cleanup (Danny Al-Gaaf, David Anderson)
+* misc coverity fixes (Danny Al-Gaaf)
+* misc coverity fixes (Danny Al-Gaaf)
+* misc: various valgrind fixes and cleanups (Danny Al-Gaaf)
+* mon: 'osd crush reweight-all' command (Sage Weil)
+* mon: add 'ceph osd rename-bucket ...' command (Loic Dachary)
+* mon: add bootstrap-rgw profile (Sage Weil)
+* mon: add max pgs per osd warning (Sage Weil)
+* mon: add noforward flag for some mon commands (Mykola Golub)
+* mon: allow adding tiers to fs pools (#10135 John Spray)
+* mon: allow full flag to be manually cleared (#9323 Sage Weil)
+* mon: clean up auth list output (Loic Dachary)
+* mon: delay failure injection (Joao Eduardo Luis)
+* mon: disallow empty pool names (#10555 Wido den Hollander)
+* mon: do not deactivate last mds (#10862 John Spray)
+* mon: do not pollute mon dir with CSV files from CRUSH check (Loic Dachary)
+* mon: drop old ceph_mon_store_converter (Sage Weil)
+* mon: fix 'ceph pg dump_stuck degraded' (Xinxin Shu)
+* mon: fix 'mds fail' for standby MDSs (John Spray)
+* mon: fix 'osd crush link' id resolution (John Spray)
+* mon: fix 'profile osd' use of config-key function on mon (#10844 Joao Eduardo Luis)
+* mon: fix *_ratio units and types (Sage Weil)
+* mon: fix JSON dumps to dump floats as flots and not strings (Sage Weil)
+* mon: fix MDS health status from peons (#10151 John Spray)
+* mon: fix caching for min_last_epoch_clean (#9987 Sage Weil)
+* mon: fix clock drift time check interval (#10546 Joao Eduardo Luis)
+* mon: fix compatset initalization during mkfs (Joao Eduardo Luis)
+* mon: fix error output for add_data_pool (#9852 Joao Eduardo Luis)
+* mon: fix feature tracking during elections (Joao Eduardo Luis)
+* mon: fix formatter 'pg stat' command output (Sage Weil)
+* mon: fix mds gid/rank/state parsing (John Spray)
+* mon: fix misc error paths (Joao Eduardo Luis)
+* mon: fix paxos off-by-one corner case (#9301 Sage Weil)
+* mon: fix paxos timeouts (#10220 Joao Eduardo Luis)
+* mon: fix stashed monmap encoding (#5203 Xie Rui)
+* mon: fix units in store stats (Joao Eduardo Luis)
+* mon: get canonical OSDMap from leader (#10422 Sage Weil)
+* mon: ignore failure reports from before up_from (#10762 Dan van der Ster, Sage Weil)
+* mon: implement 'fs reset' command (John Spray)
+* mon: improve error handling on erasure code profile set (#10488, #11144 Loic Dachary)
+* mon: improved corrupt CRUSH map detection (Joao Eduardo Luis)
+* mon: include entity name in audit log for forwarded requests (#9913 Joao Eduardo Luis)
+* mon: include pg_temp count in osdmap summary (Sage Weil)
+* mon: log health summary to cluster log (#9440 Joao Eduardo Luis)
+* mon: make 'mds fail' idempotent (John Spray)
+* mon: make pg dump {sum,pgs,pgs_brief} work for format=plain (#5963 #6759 Mykola Golub)
+* mon: new 'ceph pool ls [detail]' command (Sage Weil)
+* mon: new pool safety flags nodelete, nopgchange, nosizechange (#9792 Mykola Golub)
+* mon: new, friendly 'ceph pg ls ...' command (Xinxin Shu)
+* mon: paxos: allow reads while proposing (#9321 #9322 Joao Eduardo Luis)
+* mon: prevent MDS transition from STOPPING (#10791 Greg Farnum)
+* mon: propose all pending work in one transaction (Sage Weil)
+* mon: remove pg_temps for nonexistent pools (Joao Eduardo Luis)
+* mon: require mon_allow_pool_delete option to remove pools (Sage Weil)
+* mon: respect down flag when promoting standbys (John Spray)
+* mon: set globalid prealloc to larger value (Sage Weil)
+* mon: set {read,write}_tier on 'osd tier add-cache ...' (Jianpeng Ma)
+* mon: skip zeroed osd stats in get_rule_avail (#10257 Joao Eduardo Luis)
+* mon: validate min_size range (Jianpeng Ma)
+* mon: wait for writeable before cross-proposing (#9794 Joao Eduardo Luis)
+* mount.ceph: fix suprious error message (#10351 Yan, Zheng)
+* ms: xio: fix misc bugs (Matt Benjamin, Vu Pham)
+* msgr: async: bind threads to CPU cores, improved poll (Haomai Wang)
+* msgr: async: many fixes, unit tests (Haomai Wang)
+* msgr: async: several fixes (Haomai Wang)
+* msgr: asyncmessenger: add kqueue support (#9926 Haomai Wang)
+* msgr: avoid useless new/delete (Haomai Wang)
+* msgr: fix RESETSESSION bug (#10080 Greg Farnum)
+* msgr: fix crc configuration (Mykola Golub)
+* msgr: fix delay injection bug (#9910 Sage Weil, Greg Farnum)
+* msgr: misc unit tests (Haomai Wang)
+* msgr: new AsymcMessenger alternative implementation (Haomai Wang)
+* msgr: prefetch data when doing recv (Yehuda Sadeh)
+* msgr: simple: fix rare deadlock (Greg Farnum)
+* msgr: simple: retry binding to port on failure (#10029 Wido den Hollander)
+* msgr: xio: XioMessenger RDMA support (Casey Bodley, Vu Pham, Matt Benjamin)
+* objectstore: deprecate collection attrs (Sage Weil)
+* osd, librados: fadvise-style librados hints (Jianpeng Ma)
+* osd, librados: fix xattr_cmp_u64 (Dongmao Zhang)
+* osd, librados: revamp PG listing API to handle namespaces (#9031 #9262 #9438 David Zafman)
+* osd, mds: 'ops' as shorthand for 'dump_ops_in_flight' on asok (Sage Weil)
+* osd, mon: add checksums to all OSDMaps (Sage Weil)
+* osd, mon: send intiial pg create time from mon to osd (#9887 David Zafman)
+* osd,mon: add 'norebalance' flag (Kefu Chai)
+* osd,mon: specify OSD features explicitly in MOSDBoot (#10911 Sage Weil)
+* osd: DBObjectMap: fix locking to prevent rare crash (#9891 Samuel Just)
+* osd: EIO on whole-object reads when checksum is wrong (Sage Weil)
+* osd: add erasure code corpus (Loic Dachary)
+* osd: add fadvise flags to ObjectStore API (Jianpeng Ma)
+* osd: add get_latest_osdmap asok command (#9483 #9484 Mykola Golub)
+* osd: add misc tests (Loic Dachary, Danny Al-Gaaf)
+* osd: add option to prioritize heartbeat network traffic (Jian Wen)
+* osd: add support for the SHEC erasure-code algorithm (Takeshi Miyamae, Loic Dachary)
+* osd: allow deletion of objects with watcher (#2339 Sage Weil)
+* osd: allow recovery while below min_size (Samuel Just)
+* osd: allow recovery with fewer than min_size OSDs (Samuel Just)
+* osd: allow sparse read for Push/Pull (Haomai Wang)
+* osd: allow whiteout deletion in cache pool (Sage Weil)
+* osd: allow writes to degraded objects (Samuel Just)
+* osd: allow writes to degraded objects (Samuel Just)
+* osd: avoid publishing unchanged PG stats (Sage Weil)
+* osd: batch pg log trim (Xinze Chi)
+* osd: cache pool: ignore min flush age when cache is full (Xinze Chi)
+* osd: cache recent ObjectContexts (Dong Yuan)
+* osd: cache reverse_nibbles hash value (Dong Yuan)
+* osd: clean up internal ObjectStore interface (Sage Weil)
+* osd: cleanup boost optionals (William Kennington)
+* osd: clear cache on interval change (Samuel Just)
+* osd: do no proxy reads unless target OSDs are new (#10788 Sage Weil)
+* osd: do not abort deep scrub on missing hinfo (#10018 Loic Dachary)
+* osd: do not update digest on inconsistent object (#10524 Samuel Just)
+* osd: don't record digests for snapdirs (#10536 Samuel Just)
+* osd: drop upgrade support for pre-dumpling (Sage Weil)
+* osd: enable and use posix_fadvise (Sage Weil)
+* osd: erasure coding: allow bench.sh to test ISA backend (Yuan Zhou)
+* osd: erasure-code: encoding regression tests, corpus (#9420 Loic Dachary)
+* osd: erasure-code: enforce chunk size alignment (#10211 Loic Dachary)
+* osd: erasure-code: jerasure support for NEON (Loic Dachary)
+* osd: erasure-code: relax cauchy w restrictions (#10325 David Zhang, Loic Dachary)
+* osd: erasure-code: update gf-complete to latest upstream (Loic Dachary)
+* osd: expose non-journal backends via ceph-osd CLI (Hoamai Wang)
+* osd: filejournal: don't cache journal when not using direct IO (Jianpeng Ma)
+* osd: fix JSON output for stray OSDs (Loic Dachary)
+* osd: fix OSDCap parser on old (el6) boost::spirit (#10757 Kefu Chai)
+* osd: fix OSDCap parsing on el6 (#10757 Kefu Chai)
+* osd: fix ObjectStore::Transaction encoding version (#10734 Samuel Just)
+* osd: fix WBTHrottle perf counters (Haomai Wang)
+* osd: fix and document last_epoch_started semantics (Samuel Just)
+* osd: fix auth object selection during repair (#10524 Samuel Just)
+* osd: fix backfill bug (#10150 Samuel Just)
+* osd: fix bug in pending digest updates (#10840 Samuel Just)
+* osd: fix cancel_proxy_read_ops (Sage Weil)
+* osd: fix cleanup of interrupted pg deletion (#10617 Sage Weil)
+* osd: fix divergent entry handling on PG split (Samuel Just)
+* osd: fix ghobject_t formatted output to include shard (#10063 Loic Dachary)
+* osd: fix ioprio option (Mykola Golub)
+* osd: fix ioprio options (Loic Dachary)
+* osd: fix journal shutdown race (Sage Weil)
+* osd: fix journal wrapping bug (#10883 David Zafman)
+* osd: fix leak in SnapTrimWQ (#10421 Kefu Chai)
+* osd: fix leak on shutdown (Kefu Chai)
+* osd: fix memstore free space calculation (Xiaoxi Chen)
+* osd: fix mixed-version peering issues (Samuel Just)
+* osd: fix object age eviction (Zhiqiang Wang)
+* osd: fix object atime calculation (Xinze Chi)
+* osd: fix object digest update bug (#10840 Samuel Just)
+* osd: fix occasional peering stalls (#10431 Sage Weil)
+* osd: fix ordering issue with new transaction encoding (#10534 Dong Yuan)
+* osd: fix osd peer check on scrub messages (#9555 Sage Weil)
+* osd: fix past_interval display bug (#9752 Loic Dachary)
+* osd: fix past_interval generation (#10427 #10430 David Zafman)
+* osd: fix pgls filter ops (#9439 David Zafman)
+* osd: fix recording of digest on scrub (Samuel Just)
+* osd: fix scrub delay bug (#10693 Samuel Just)
+* osd: fix scrub vs try-flush bug (#8011 Samuel Just)
+* osd: fix short read handling on push (#8121 David Zafman)
+* osd: fix stderr with -f or -d (Dan Mick)
+* osd: fix transaction accounting (Jianpeng Ma)
+* osd: fix watch reconnect race (#10441 Sage Weil)
+* osd: fix watch timeout cache state update (#10784 David Zafman)
+* osd: fix whiteout handling (Sage Weil)
+* osd: flush snapshots from cache tier immediately (Sage Weil)
+* osd: force promotion of watch/notify ops (Zhiqiang Wang)
+* osd: handle no-op write with snapshot (#10262 Sage Weil)
+* osd: improve idempotency detection across cache promotion/demotion (#8935 Sage Weil, Samuel Just)
+* osd: include activating peers in blocked_by (#10477 Sage Weil)
+* osd: jerasure and gf-complete updates from upstream (#10216 Loic Dachary)
+* osd: journal: check fsync/fdatasync result (Jianpeng Ma)
+* osd: journal: fix alignment checks, avoid useless memmove (Jianpeng Ma)
+* osd: journal: fix hang on shutdown (#10474 David Zafman)
+* osd: journal: fix header.committed_up_to (Xinze Chi)
+* osd: journal: fix journal zeroing when direct IO is enabled (Xie Rui)
+* osd: journal: initialize throttle (Ning Yao)
+* osd: journal: misc bug fixes (#6003 David Zafman, Samuel Just)
+* osd: journal: update committed_thru after replay (#6756 Samuel Just)
+* osd: keyvaluestore: cleanup dead code (Ning Yao)
+* osd: keyvaluestore: fix getattr semantics (Haomai Wang)
+* osd: keyvaluestore: fix key ordering (#10119 Haomai Wang)
+* osd: keyvaluestore_dev: optimization (Chendi Xue)
+* osd: limit in-flight read requests (Jason Dillaman)
+* osd: log when scrub or repair starts (Loic Dachary)
+* osd: make misdirected op checks robust for EC pools (#9835 Sage Weil)
+* osd: memstore: fix size limit (Xiaoxi Chen)
+* osd: misc FIEMAP fixes (Ma Jianpeng)
+* osd: misc cleanup (Xinze Chi, Yongyue Sun)
+* osd: misc optimizations (Xinxin Shu, Zhiqiang Wang, Xinze Chi)
+* osd: misc scrub fixes (#10017 Loic Dachary)
+* osd: new 'activating' state between peering and active (Sage Weil)
+* osd: new optimized encoding for ObjectStore::Transaction (Dong Yuan)
+* osd: optimize Finisher (Xinze Chi)
+* osd: optimize WBThrottle map with unordered_map (Ning Yao)
+* osd: optimize filter_snapc (Ning Yao)
+* osd: preserve reqids for idempotency checks for promote/demote (Sage Weil, Zhiqiang Wang, Samuel Just)
+* osd: proxy read support (Zhiqiang Wang)
+* osd: proxy reads during cache promote (Zhiqiang Wang)
+* osd: remove dead locking code (Xinxin Shu)
+* osd: remove legacy classic scrub code (Sage Weil)
+* osd: remove unused fields in MOSDSubOp (Xiaoxi Chen)
+* osd: removed some dead code (Xinze Chi)
+* osd: replace MOSDSubOp messages with simpler, optimized MOSDRepOp (Xiaoxi Chen)
+* osd: restrict scrub to certain times of day (Xinze Chi)
+* osd: rocksdb: fix shutdown (Hoamai Wang)
+* osd: store PG metadata in per-collection objects for better concurrency (Sage Weil)
+* osd: store whole-object checksums on scrub, write_full (Sage Weil)
+* osd: support for discard for journal trim (Jianpeng Ma)
+* osd: use FIEMAP_FLAGS_SYNC instead of fsync (Jianpeng Ma)
+* osd: verify kernel is new enough before using XFS extsize ioctl, enable by default (#9956 Sage Weil)
+* pybind: fix memory leak in librados bindings (Billy Olsen)
+* pyrados: add object lock support (#6114 Mehdi Abaakouk)
+* pyrados: fix misnamed wait_* routings (#10104 Dan Mick)
+* pyrados: misc cleanups (Kefu Chai)
+* qa: add large auth ticket tests (Ilya Dryomov)
+* qa: fix mds tests (#10539 John Spray)
+* qa: fix osd create dup tests (#10083 Loic Dachary)
+* qa: ignore duplicates in rados ls (Josh Durgin)
+* qa: improve hadoop tests (Noah Watkins)
+* qa: many 'make check' improvements (Loic Dachary)
+* qa: misc tests (Loic Dachary, Yan, Zheng)
+* qa: parallelize make check (Loic Dachary)
+* qa: reorg fs quota tests (Greg Farnum)
+* qa: tolerate nearly-full disk for make check (Loic Dachary)
+* rados: fix put of /dev/null (Loic Dachary)
+* rados: fix usage (Jianpeng Ma)
+* rados: parse command-line arguments more strictly (#8983 Adam Crume)
+* rados: use copy-from operation for copy, cppool (Sage Weil)
+* radosgw-admin: add replicalog update command (Yehuda Sadeh)
+* rbd-fuse: clean up on shutdown (Josh Durgin)
+* rbd-fuse: fix memory leak (Adam Crume)
+* rbd-replay-many (Adam Crume)
+* rbd-replay: --anonymize flag to rbd-replay-prep (Adam Crume)
+* rbd: add 'merge-diff' function (MingXin Liu, Yunchuan Wen, Li Wang)
+* rbd: allow v2 striping parameters for clones and imports (Jason Dillaman)
+* rbd: fix 'rbd diff' for non-existent objects (Adam Crume)
+* rbd: fix buffer handling on image import (#10590 Jason Dillaman)
+* rbd: fix error when striping with format 1 (Sebastien Han)
+* rbd: fix export for image sizes over 2GB (Vicente Cheng)
+* rbd: fix formatted output of image features (Jason Dillaman)
+* rbd: leave exclusive lockin goff by default (Jason Dillaman)
+* rbd: updat eman page (Ilya Dryomov)
+* rbd: update init-rbdmap to fix dup mount point (Karel Striegel)
+* rbd: use IO hints for import, export, and bench operations (#10462 Jason Dillaman)
+* rbd: use rolling average for rbd bench-write throughput (Jason Dillaman)
+* rbd_recover_tool: RBD image recovery tool (Min Chen)
+* rgw: S3-style object versioning support (Yehuda Sadeh)
+* rgw: add location header when object is in another region (VRan Liu)
+* rgw: change multipart upload id magic (#10271 Yehuda Sadeh)
+* rgw: check keystone auth for S3 POST requests (#10062 Abhishek Lekshmanan)
+* rgw: check timestamp on s3 keystone auth (#10062 Abhishek Lekshmanan)
+* rgw: conditional PUT on ETag (#8562 Ray Lv)
+* rgw: create subuser if needed when creating user (#10103 Yehuda Sadeh)
+* rgw: decode http query params correction (#10271 Yehuda Sadeh)
+* rgw: don't overwrite bucket/object owner when setting ACLs (#10978 Yehuda Sadeh)
+* rgw: enable IPv6 for civetweb (#10965 Yehuda Sadeh)
+* rgw: extend replica log API (purge-all) (Yehuda Sadeh)
+* rgw: fail S3 POST if keystone not configured (#10688 Valery Tschopp, Yehuda Sadeh)
+* rgw: fix If-Modified-Since (VRan Liu)
+* rgw: fix XML header on get ACL request (#10106 Yehuda Sadeh)
+* rgw: fix bucket removal with data purge (Yehuda Sadeh)
+* rgw: fix content length check (#10701 Axel Dunkel, Yehuda Sadeh)
+* rgw: fix content-length update (#9576 Yehuda Sadeh)
+* rgw: fix disabling of max_size quota (#9907 Dong Lei)
+* rgw: fix error codes (#10334 #10329 Yehuda Sadeh)
+* rgw: fix incorrect len when len is 0 (#9877 Yehuda Sadeh)
+* rgw: fix object copy content type (#9478 Yehuda Sadeh)
+* rgw: fix partial GET in swift (#10553 Yehuda Sadeh)
+* rgw: fix replica log indexing (#8251 Yehuda Sadeh)
+* rgw: fix shutdown (#10472 Yehuda Sadeh)
+* rgw: fix swift metadata header name (Dmytro Iurchenko)
+* rgw: fix sysvinit script when rgw_socket_path is not defined (#11159 Yehuda Sadeh, Dan Mick)
+* rgw: fix user stags in get-user-info API (#9359 Ray Lv)
+* rgw: include XML ns on get ACL request (#10106 Yehuda Sadeh)
+* rgw: index swift keys appropriately (#10471 Yehuda Sadeh)
+* rgw: make sysvinit script set ulimit -n properly (Sage Weil)
+* rgw: misc fixes (#10307 Yehuda Sadeh)
+* rgw: only track cleanup for objects we write (#10311 Yehuda Sadeh)
+* rgw: pass civetweb configurables through (#10907 Yehuda Sadeh)
+* rgw: prevent illegal bucket policy that doesn't match placement rule (Yehuda Sadeh)
+* rgw: remove multipart entries from bucket index on abort (#10719 Yehuda Sadeh)
+* rgw: remove swift user manifest (DLO) hash calculation (#9973 Yehuda Sadeh)
+* rgw: respond with 204 to POST on containers (#10667 Yuan Zhou)
+* rgw: return timestamp on GET/HEAD (#8911 Yehuda Sadeh)
+* rgw: reuse fcgx connection struct (#10194 Yehuda Sadeh)
+* rgw: run radosgw as apache with systemd (#10125 Loic Dachary)
+* rgw: send explicit HTTP status string (Yehuda Sadeh)
+* rgw: set ETag on object copy (#9479 Yehuda Sadeh)
+* rgw: set length for keystone token validation request (#7796 Yehuda Sadeh, Mark Kirkwood)
+* rgw: support X-Storage-Policy header for Swift storage policy compat (Yehuda Sadeh)
+* rgw: support multiple host names (#7467 Yehuda Sadeh)
+* rgw: swift: dump container's custom metadata (#10665 Ahmad Faheem, Dmytro Iurchenko)
+* rgw: swift: support Accept header for response format (#10746 Dmytro Iurchenko)
+* rgw: swift: support for X-Remove-Container-Meta-{key} (#10475 Dmytro Iurchenko)
+* rgw: tweak error codes (#10329 #10334 Yehuda Sadeh)
+* rgw: update bucket index on attr changes, for multi-site sync (#5595 Yehuda Sadeh)
+* rgw: use \r\n for http headers (#9254 Yehuda Sadeh)
+* rgw: use gc for multipart abort (#10445 Aaron Bassett, Yehuda Sadeh)
+* rgw: use new watch/notify API (Yehuda Sadeh, Sage Weil)
+* rpm: misc fixes (Key Dreyer)
+* rpm: move rgw logrotate to radosgw subpackage (Ken Dreyer)
+* systemd: better systemd unit files (Owen Synge)
+* sysvinit: fix race in 'stop' (#10389 Loic Dachary)
+* test: fix bufferlist tests (Jianpeng Ma)
+* tests: ability to run unit tests under docker (Loic Dachary)
+* tests: centos-6 dockerfile (#10755 Loic Dachary)
+* tests: improve docker-based tests (Loic Dachary)
+* tests: unit tests for shared_cache (Dong Yuan)
+* udev: fix rules for CentOS7/RHEL7 (Loic Dachary)
+* use clock_gettime instead of gettimeofday (Jianpeng Ma)
+* vstart.sh: set up environment for s3-tests (Luis Pabon)
+* vstart.sh: work with cmake (Yehuda Sadeh)
+
+
+
+
+
v0.93
=====