doc/dev: integrate testing into the narrative

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7941 from SUSE/wip-teuos-intro

doc: detailed description of bugfixing workflow

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: detailed description of bugfixing workflow

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7933 from ceph/wip-doc-dist-ceph-detect-init

doc: add ceph-detect-init(8) source to dist tarball

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7937 from saru95/patch--1

ceph_daemon.py: Resolved ImportError to work with python3

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7936 from ceph/wip-fix-debian-build

debian/changelog: Remove stray 'v' in version

ceph_daemon.py: Resolved ImportError to work with python3

Signed-off-by: Sarthak Munshi <sarthakmunshi@gmail.com>

debian/changelog: Remove stray 'v' in version

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #7918 from ceph/wip-rgw-payer-yehuda

rgw: link payer info to usage logging

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

doc: add ceph-detect-init(8) source to dist tarball

Prior to this commit, the tarball from "make dist" did not include the
ceph-detect-init(8) man page rST source.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #7929 from yehudasa/wip-submodules-fix

submodules: revert an accidental change

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #7910 from ceph/wip-14955

buffer: increment history alloc as well in raw_combined

Reviewed-by: Sage Weil <sage@redhat.com>

submodules: revert an accidental change

revert changes that went in in commit f1ac0de5bfd974258c257470feae5ff03054873e.
This affects ceph-erasure-code-corpus, civetweb

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7916 from ceph/wip-14975

rgw: adjust error code when bucket does not exist in copy operation
rgw: don't override error when initializing zonegroup

Fixes: #14975
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: avoid showing payer when payer == owner

If the payer matches the owner it shows the owner only.

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

Merge pull request #7786 from ceph/wip-rgw-indexless

rgw: indexless buckets (Yehuda Sadeh)

 - can define a policy, for which buckets are indexless
 - users can then create buckets under the specified placement target
 - indexless buckets will not be synced across zones
 - does not work with (s3) versioned buckets

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: indexless buckets

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: configurable index type

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't override error when initializing zonegroup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adjust error code when bucket does not exist in copy operation

Fixes: #14975
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7764 from rzarzynski/wip-rgw-etag-dlo

rgw: multiple improvements regarding etag calculation for SLO/DLO of Swift API.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

buffer: increment history alloc as well in raw_combined

Introduced in 73dcd26fc8798e09d93520c583d63edb28ffc631

Fixes: #14955
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge remote-tracking branch 'gh/jewel'

v10.0.4

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7897 from branch-predictor/bp-bench-errors-fatal

common/obj_bencher.cc: make verify error fatal

Reviewed-by: Loic Dachary <ldachary@redhat.com>

common/obj_bencher.cc: make verify error fatal

When run without "--no-verify", all verification errors are noted,
but they are not forwarded/reported anywhere else but to cerr, which
will cause automated testing to ignore them. Make seq_read_bench and
rand_read_bench return -EIO on any verification error which will,
in turn, return it back to caller.

Fixes: #14971
Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #7757 from ukernel/jewel-14684

mon/MDSMonitor.cc: properly note beacon when health metrics changes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #7882 from branch-predictor/bp-unconfuse-health-1

mon: unconfuse object count skew message

Reviewed-by: Sage Weil <sage@redhat.com>

PGMonitor: unconfuse object count skew message

"Pool <pool> has too few pgs" is okay assuming it does not take other
pools into account. And since it does, it is confusing in the following
scenario:

1. Create two pools, one with small pg count and one with large
   pg count
2. Put a whole lot of objects in smaller pool, resulting in "too few
   pgs" warning on that pool, which is expected behavior.
3. Put a whole lot of objects in larger pool, warning goes away.
   Suddenly smaller pool has plenty of PGs?

Current message suggests adding more nodes (or PGs) to pool, when
actually it's warning about significantly more objects in that
particular pool than in the other pools.

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #7898 from oritwas/wip-cmake-add-fs_types

cmake: add common/fs_types.cc to libcommon

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #7881 from aclamk/ceph-df-used-fix

mon: fix calculation of %USED

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7892 from majianpeng/bluestore-leak-unwritten-extent

osd: bluestore: fix check for write falling within the same extent

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7886 from stiopaa1/blueStoreOverrides

osd: bluestore: add 'override' to virtual functions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7895 from majianpeng/bluestore-trim-overlay

os/bluestore/BlueStore: Don't leak trim overlay data before write.

os/bluestore/BlueStore: Don't leak trim overlay data before write.

Suppose: bluestore_overlay_max_length=bluestore_min_alloc_size;
bluestore_overlay_max = 2;
For the following ops:
write(off=0, len=4096) --->write into overlay
write(off=4096, len=4096)-->write into overlay
write(off=0, len=bluestore_min_alloc_size)-->because overlay_map.size()
>=2, it allocate a extent.

It should trim overlay data(0,4096) &(4096, 4096),and then write(0,
bluestore_min_alloc_size).
But the original code don't trim overlay data.
This make the later read data is orignal data rather that new data.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

cmake: add common/fs_types.cc to libcommon

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

os/bluestore/BlueStore: Fix bug when calc offset & end whether locate in the a extent.

Suppose: bluestore_overlay_max_length == bluestore_min_alloc_size
The orignal code which calc content of written whether locate in a
extent:
(offset / min_alloc_size) == (offset + length) /min_alloc_size
This will make the case which offset=0 & length =min_alloc_size locate
in the different extent.
In fact, this content is in the same extent.

Change end = offset + length - 1 make work.

Fixes: #14954
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/obj_bencher.cc: use more readable constant instead of magic number

When clean_up_slow() fails, it returns "-5" which is equal to -EIO.
Change it in source, so it's not confusing for someone who does not
remember all error codes (functionality remains the same).

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #7098 from liewegas/wip-file-layout

mds, client: add namespace to file_layout_t (previously ceph_file_layout)

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Bluestore.h:Added 'override' to virtual functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #7849 from ceph/wip-cmake

Wip cmake

cmake: simplified heap_profiler_objs dependencies

The heap_profiler object library only needs to be
added to mon,mds, and osd instead of the numerous
targets it is part of

Signed-off-by: Ali Maredia <amaredia@redhat.com>

client/Inode: fix has_dir_layout

Compare to default file_layout_t() -- not zeroed memory.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7867 from SUSE/wip-14940

RPM: move %post(un) ldconfig calls to ceph-base

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #7869 from rzarzynski/wip-rgw-aws4-puthash

rgw: calculate payload hash in RGWPutObj_ObjStore only when necessary.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge pull request #7612 from liewegas/wip-buffer-combined

buffer: raw_combined allocations buffer and ref count together

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #7690 from xiexingguo/xxg-wip-14804

common/TrackedOp: fix inaccurate counting for slow requests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7704 from xiexingguo/xxg-wip-14814

osd: filestore: fix wrong scope of result code for error cases during mkfs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7812 from xiexingguo/xxg-wip-14872

osd: fix wrong return type of find_osd_on_ip()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7815 from branch-predictor/bp-conditional-crc

msg/async: don't calculate msg header crc when not needed

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7711 from xiexingguo/xxg-wip-14817

osd: filestore: fix result code overwritten for clone

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7790 from tchaikov/wip-log-segv

log: segv in a portable way

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7842 from xiexingguo/xxg-wip-14921

osd: fix overload of '==' operator for pg_stat_t

Reviewed-by: Sage Weil <sage@redhat.com>

[MON] Fixed calculation of %USED. Now it is shows (space used by all replicas)/(raw space available on OSDs). Before it was (size of pool)/(raw space available on OSDs).

Signed-off-by: Adam Kupczyk akupczyk@mirantis.com

Merge pull request #7390 from badone/wip-13829

common: Allow OPT_INT settings with negative values

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #7854 from yehudasa/wip-authors-update

AUTHORS: update email

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7846 from dillaman/wip-14551

rbd: update default image features

Conflicts:
PendingReleaseNotes (trivial)

Merge pull request #7841 from dillaman/wip-14550

rbd: deprecate image format 1

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7839 from dillaman/wip-14809

librbd: remove last synchronous librados calls from open/close state machine

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7785 from dillaman/wip-14822

librbd: journal replay needs to support re-executing maintenance ops

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7776 from dillaman/wip-librbd-journal-tweaks

journal: improve commit position tracking

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cmake: Turned LTTng OFF

This is temporary, until a way to stop repetitive
unecessary generation of headerfiles is figured out.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

RPM: move scriptlets from ceph to ceph-base

This addresses the following RPMLINT error:

ceph-base.x86_64: E: library-without-ldconfig-postun (Badness:
300) /usr/lib64/libosd_tp.so.1.0.0
ceph-base.x86_64: E: library-without-ldconfig-postun (Badness:
300) /usr/lib64/libos_tp.so.1.0.0
This package contains a library and provides no %postun scriptlet
containing a call to ldconfig.

ceph-base.x86_64: E: library-without-ldconfig-postin (Badness:
300) /usr/lib64/libosd_tp.so.1.0.0
ceph-base.x86_64: E: library-without-ldconfig-postin (Badness:
300) /usr/lib64/libos_tp.so.1.0.0
This package contains a library and provides no %post scriptlet
containing a call to ldconfig.

http://tracker.ceph.com/issues/14940 Fixes: #14940

Signed-off-by: Nathan Cutler <ncutler@suse.com>

AUTHORS: update email

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: calculate payload hash in RGWPutObj_ObjStore only when necessary.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #7767 from athanatos/wip-sam-journal-throttle-4

osd: filestore: restructure journal and op queue throttling

Reviewed-by: Samuel Just <sjust@redhat.com>

mds: deny access to file with pool_ns layout if feature is missing

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: fix inodestat size calculation

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: properly purge backtrace object

backtrace objects are always in default namespace

Signed-off-by: Yan, Zheng <zyan@redhat.com>

messages/MClientCaps: fix decoding

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds/Server: fix whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

mds/Locker: fix Session leak on access denial

Signed-off-by: Sage Weil <sage@redhat.com>

fix init of new fields in MCLientCaps

mds/MDSMap: add file layout v2 incompat feature

The existing MDSMonitor machinations already add this to the MDSMap to
boot out old MDS's.

Signed-off-by: Sage Weil <sage@redhat.com>

mds: deny access to directories with pool_ns layouts if feature is missing

If the client feature is missing, given them EIO.  Change check_access to
return an error code to distinguish this from EACCES.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MDSMonitor: prevent pool 0 from being used as a data pool

Pool 0 means no change or default in the legacy ceph_file_layout in the
layout ioctl and file create arguments. Prevent it from being used to avoid
putting users in an awkward situation later.

Signed-off-by: Sage Weil <sage@redhat.com>

mds: don't break compatibility of MClientCaps/MClientReply

Signed-off-by: Yan, Zheng <zyan@redhat.com>

fs_types: file_layout_t: convert pool -1 (undefined) to 0 in legacy encoding

Old code assumes that fl_pg_pool == 0 means the pool is not defined, while
file_layout_t uses -1.  Translate between the two.

Note that this means a valid file_layout_t with pool_id == 0 cannot be
accurately translated to a legacy file_layout_t.  That is somewhat
unavoidable, and should not be a problem since real clusters create 'rbd'
as pool 0 and it does not use any file layouts.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/fs/misc/layout_vxattrs: test layout.pool_namespace

Signed-off-by: Sage Weil <sage@redhat.com>

client: expose layout.pool_namespace vxattr

Signed-off-by: Sage Weil <sage@redhat.com>

mds: allow adjustment of layout.pool_namespace vxattr

Signed-off-by: Sage Weil <sage@redhat.com>

struct ceph_file_layout -> file_layout_t

- drop the global
- do not memset!
- encode with features
- field names are different
- use get_period() method where appropriate
- fix is layout empty checks

Signed-off-by: Sage Weil <sage@redhat.com>

mds: make inode_t, old_inode_t featureful

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MClientCaps: avoid fixed struct for encoding

At least for new clients.

Signed-off-by: Sage Weil <sage@redhat.com>

mds: encode inode reply information directly

Drop the ceph_mds_reply_inode struct and encode directly.  Note
that we already decode directly to InodeStat in MClientReply.h.

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MClientReply: stop using ceph_mds_reply_inode struct to unpack

Decode directly.

Signed-off-by: Sage Weil <sage@redhat.com>

mds: pass features to various CInode encoders and helpers

We'll need this shortly.

Signed-off-by: Sage Weil <sage@redhat.com>

mds/MDSMap: cache up_features; fix logic

& these, not | them.

Signed-off-by: Sage Weil <sage@redhat.com>

libradosstriper: move default stripe parameters inline

The extern is going away.

Signed-off-by: Sage Weil <sage@redhat.com>

include/encoding: featureful encoder for list of shared_ptrs

Signed-off-by: Sage Weil <sage@redhat.com>

include/compact_map: featureful encoders, too

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-dencoder: TYPE_FEATUREFUL_NOCOPY

Signed-off-by: Sage Weil <sage@redhat.com>

mds: pass features to LogEvents, EMetaBlob

Signed-off-by: Sage Weil <sage@redhat.com>

mds,mon: include features in beacon messages, MDSMap

Signed-off-by: Sage Weil <sage@redhat.com>

mds/MDSMap: add features to MDSMap's mds_info_t

Signed-off-by: Sage Weil <sage@redhat.com>

add fs_types.h, with file_layout_t

Signed-off-by: Sage Weil <sage@redhat.com>

declare Formatter for everyone

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/fs/misc/layout_vxattr.sh: detect data pool name

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7823 from SUSE/wip-14906

packaging: add lsb-release dependency for SUSE and Debian

Reviewed-by: David Disseldorp <ddiss@suse.de>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>