mds: update CInode::oldest_snap during migration

Fixes: #12105
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #4981 from ceph/wip-fedora-build-fixes

Fix various issues with fedora (f21+) builds

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #4898 from SUSE/wip_ceph_spec_sharutils

ceph.spec.in:BuildRequires sharutils

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

ceph.spec.in:BuildRequires sharutils

The uudecode binary is used to build Java-related components, and
uudecode is provided by the sharutils package on all supported
RPM platforms. When building with "--without=cephfs_java",
sharutils is not needed.

Thanks to Nathan Cutler <ncutler@suse.cz> for going into the
details with me.

On OBS without this patch we get the error message:

[  170s] -----------------------------------------------------------------
[  170s] ----- building ceph.spec (user abuild)
[  170s] -----------------------------------------------------------------
[  170s] -----------------------------------------------------------------
[  170s] + exec rpmbuild -ba --define '_srcdefattr (-,root,root)' --nosignature --define 'disturl obs://build.opensuse.org/home:osynge:ceph:wip:wip_obs_fedora/Fedora_20/c0bbbc1e62228ca956ac3d367edc4fba-master' /home/abuild/rpmbuild/SOURCES/ceph.spec
[  170s] error: Failed build dependencies:
[  170s]    sharutils is needed by ceph-1:2+git.1435043747.c1bd02c-1.1.x86_64

With this patch we can build fedora 22 and fedora 20 rpms fine.

Signed-off-by: Owen Synge <osynge@suse.com>

Merge pull request #5042 from ceph/wip-final-kw

mds/Migrator: don't use c++11 kw 'final' as var name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5066 from ceph/wip-gitignore-add-ceph_perf_msgr

src/.gitignore: add ceph_perf_msgr_{server,client}

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4840 from tchaikov/wip-11833-crush-dump-tree

mon: add an "osd crush dump tree" command

Reviewed-by: Min Chen <minchen@ubuntukylin.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

src/.gitignore: add ceph_perf_msgr_{server,client}

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4965 from xinxinsh/wip-12025

mon/OSDMonitor : error out if pgp_num > pg_num

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4923 from dachary/wip-make-check-verbosity

tests: display the output of failed make check runs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5032 from guangyy/wip-cleanup

osd: remove duplicate checks

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5064 from vuhuong/wip-xio

xio: sync works

Reviewed-by: Kefu Chai <kchai@redhat.com>

move pgp_num, pg_num check to prepare_new_pool method

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

xio: fix to work with commit 626360aa

Fix xio to work with commit 626360aa
"msg, ceph_osd: Support feature bits for all message type's local connection"

Signed-off-by: Vu Pham <vu@mellanox.com>

xio: fix to work with the merge of pull request #4707

fix xio to work with the merge of pull request
" Removed unnecessary inclusion of iostream in several files #4707"

Signed-off-by: Vu Pham <vu@mellanox.com>

xio: sync to work with accellio v1.4

latest master's HEAD - tag v1-4

Signed-off-by: Vu Pham <vu@mellanox.com>

Merge pull request #4747 from ceph/wip-damaged-fixes

MDS metadata damage handling

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5054 from SUSE/wip-release-notes-syntax

doc/release-notes.rst: add missing word to complete sentence

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/release-notes.rst: add missing word to complete sentence

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #4792 from kylinstorage/wip-writeback-throttling-for-cache-tiering

Wip writeback throttling for cache tiering

This patch is to do write back throttling for cache tiering, which is similar to what the Linux kernel does for  page cache write back.  A paramter 'cache_target_dirty_high_ratio'  (default 0.6) is introduced as the high speed flushing threshold, while leave the 'cache_target_dirty_ratio' (default 0.4) to represent the low speed  threshold. The flush speed is controlled by limiting the parallelism of flushing. The maximum parallelism under low speed is half of the parallelism under high speed. If there is at least one PG such that the dirty ratio beyond the high threshold, full speed mode is entered; If there is no PG such that dirty ratio beyond the low threshold, idle mode is entered; In other cases, slow speed mode is entered.

Signed-off-by: Mingxin Liu <mingxinliu@ubuntukylin.com>
Reviewed-by: Li Wang <liwang@ubuntukylin.com>
Suggested-by: Nick Fisk <nick@fisk.me.uk>
Tested-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4290 from XinzeChi/wip-filestore-lat

osd: dout more log info to track io request

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5049 from ceph/wip-librbd-reduce-tests

librbd: reduce unit test execution time

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3973 from XinzeChi/wip-scrub-pending

osd: check pending or active scrub before sched_scrub

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4847 from ceph/wip-11661

osd: Improve feature bit handling and remove ancient checking including CHUNKY_SCRUB

Reviewed-by: Sage Weil <sweil@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

tests: reduce duplicate librbd unit test case coverage

The test run for RBD_FEATURES=13 is fully covered by
RBD_FEATURES=45 and the pre deep-flatten cases are covered
by RBD_FEATURES=0 and 1.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

lockdep: do not automatically collect all backtraces

It is expensive to collect backtraces every time a lock is
checked in order to provide cycle backtraces.  The backtraces
can be forced on for specific locks or globally via the new
config option "lockdep_force_backtrace".

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5041 from oritwas/wip-12095

rgw: force content_type for swift bucket stats request

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

tests: move librbd valgrind test to teuthology

This test can take up to 20 minutes to execute, which is an order of
magnitude longer than other unit tests.  This is slowing down the
build bots.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: force content_type for swift bucket stats request

Fixes: 12095
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

mds/Migrator: don't use c++11 kw 'final' as var name

If for no other reason than that it looks weird
in an editor.

Signed-off-by: John Spray <john.spray@redhat.com>

osd: remove duplicate checks

Within pg_has_reset_since, it checkes if the PG is being deleted or not,
no need a duplicate check

Signed-off-by: Guang Yang <yguang@yahoo-inc.com>

ceph_osd: Add required feature bits related to this branch to osd_required mask

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: CEPH_FEATURE_INDEP_PG_MAP feature now required

Feature present since at least the Dumpling release.
A later commit will add it to the osd_required mask

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: CEPH_FEATURE_OSD_PACKED_RECOVERY feature now required

Feature present since at least the Dumpling release.
A later commit will add it to the osd_required mask

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: CEPH_FEATURE_RECOVERY_RESERVATION feature now required

Feature present since at least the Dumpling release.
A later commit will add it to the osd_required mask

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: CEPH_FEATURE_BACKFILL_RESERVATION feature now required

Feature present since at least the Dumpling release.
A later commit will add it to the osd_required mask

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: CEPH_FEATURE_CHUNKY_SCRUB feature now required

Feature present since at least the Dumpling release.
A later commit will add it to the osd_required mask

Fixes: #11661
Signed-off-by: David Zafman <dzafman@redhat.com>

msg, ceph_osd: Support feature bits for all message type's local connection

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add tracking of acting_features and upacting_features

Use most appropriate feature intersection depending on context

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix peer_features to include self

This should have no practical effect unless we could have
features getting turned off in a later release, since we
can assume any features being checked for are supported locally.

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #5019 from zhouyuan/qemu_git_url

qa: Use public qemu repo

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4848 from leseb/doc-openstack-kilo

doc: update openstack and rgw keystone

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5012 from Arshanes/master

doc: Fixed the picture on page http://ceph.com/docs/master/cephfs/

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Fixed the picture on page http://ceph.com/docs/master/cephfs/

Fixes: #9174
Signed-off-by Valentin Thomas <valentin.arshanes.thomas@gmail.com>

Merge pull request #4918 from SUSE/wip_ceph_spec_directories_not_owned_by_a_package

ceph.spec.in: fix:Add missing directories breaking suse rpm build

Reviewed-by: Boris Ranto <branto@redhat.com>

ceph.spec.in: fix:Add missing directories breaking build

SUSE builds on OBS are failing with the missing dir entries:

    /usr/share/ceph
    /usr/lib/ceph

On suse these correspond to:

    %dir %{_datarootdir}/ceph/
    %dir %{_libexecdir}/ceph/

Signed-off-by: Owen Synge <osynge@suse.com>

Merge pull request #4969 from ceph/wip-frag-asok

mds: add dirfrag split/merge asok commands

Merge pull request #5025 from majianpeng/librbd-remove-iohint

librbd: If objectmap tell object exist, don't send alloc_hint w/ write command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: If objectmap tell object exist, don't send alloc_hint w/ write command.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #4929 from wonzhq/tracked-op

common: tracked op fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4249 from wonzhq/rmattr

osd: return with ENOENT if object doesn't exist during rmattr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5002 from SUSE/wip_fixes_to_rcceph

Fixes to rcceph script

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4983 from yuyuyu101/wip-setallochint-option

librbd: Add option to allow disabling issuing alloc hint

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4993 from ceph/wip-12064

mon: only send MMonMetadata to peer mons that support it

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #5005 from maxime1992/master

doc: Document include/str_list.h

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph.spec.in: rbd-replay-prep is not being built on f21+

Signed-off-by: Boris Ranto <branto@redhat.com>

configure.ac: Fix junit4.jar detection

If a system contains older (3) or later (5) release of junit.jar that is
default in the system, the auto-detection might find it before it finds
junit4.jar. This commit fixes that issue by always preferring the
junit4.jar library.

Signed-off-by: Boris Ranto <branto@redhat.com>

ceph.spec.in: Require git distro-wide

Git is not a SUSE-only dependency, the build process itself requires it.

Signed-off-by: Boris Ranto <branto@redhat.com>

doc: Document include/str_list.h
Fixes: #12050.
Signed-off-by: Claire MASSOT <claire.massot93@gmail.com>
Signed-off-by: Jordan DORNE <jordan.dorne@gmail.com>
Signed-off-by: Kévin CARADANT <kevin.caradant@gmail.com>
Signed-off-by: Gabriel SENTUCQ <perso@kazhord.fr>
Signed-off-by: Maxime ROBERT <maxime.robert1992@gmail.com>

Merge pull request #5017 from JordanDorne/master

doc: Replace requries with requires in python API doc

Reviewed-by: Loic Dachary <ldachary@redhat.com>

qa: Use public qemu repo

This would allow some external tests outside of sepia lab

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

doc: Replace requries with requires in python API doc
Fixes: #12076
Signed-off-by: Jordan DORNE <jordan.dorne@gmail.com>

Merge pull request #4989 from ltoto2/master

 doc: Adding a link to documentation on admin socket

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Adding a link to documentation on admin socket

Fixes: #11845
Signed-off-by: Thomas Laumondais <thomas.laumondais@gmail.com>

Bug fix to ceph systemV compatability script.

Was failing with more than one OSD / MON deamon on a single node.
Fixes suse bugzilla #927862

Signed-off-by: Owen Synge <osynge@suse.com>

Merge pull request #5011 from robin974/master

doc: Fixes a missing dot

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5006 from Claire1293/master

doc: dev/differences-from-posix

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4987 from Aeryax/master

doc: Replaced "disk or drive" by "disk" in ceph-disk.rs

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Fixes a missing dot

Fixes: #12072
Signed-off-by: Robin Tang <robintang974@gmail.com>

Merge pull request #5008 from jr31/master

doc: Unify ID format

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5007 from Cleminou/master

doc: Fixes a missing word in a printed string

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: dev/differences-from-posix
Fixes: #11108
Signed-off-by: Claire MASSOT <claire.massot93@gmail.com>

doc: Unify ID format

Fixes: #12071
Signed-off-by: Jean-Rémi Deveaux <jeanremi.deveaux@gmail.com>

doc: Fixes a missing word in a printed string

Fixes: #11106
Signed-off-by: Clement Lebrun <clement.lebrun.31@gmail.com>

Merge pull request #5004 from arthurgorjux/master

doc: Remove `` before and after 'ceph -s' in titles

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Replaced "disk or drive" by "disk" in ceph-disk.rs
Fixes: #12062
Signed-off-by: Gabriel SENTUCQ <perso@kazhord.fr>

Merge pull request #5003 from Oxygen31/master

doc : Change of "create" in "add"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4954 from JevonQ/add-iov-support-in-libcephfs

libcephfs: add ceph_p{read,write}v

Reviewed-by: John Spray <john.spray@redhat.com>

doc: Remove `` before and after 'ceph -s' in titles
Fixes: #12059
Signed-off-by: Arthur Gorjux <arthurgorjux@gmail.com>

doc : Change of "ceph-deploy mon create {ceph-node}" with "ceph-deploy mon add {ceph-node}" in "Quick-ceph-deploy.rst"

Fixes : #10293

Signed-off-by : Germain CHIPAUX <germain.chipaux@gmail.com>

Merge pull request #4986 from nicolasyong/master

doc: Modification of a sentence

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Modification of a sentence.

Fixes : #12057

Signed-off-by: Nicolas Yong <nicolas.yong93@gmail.com>

Merge pull request #4988 from kevincaradant/master

doc: ceph-deploy man page: typo

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Fixes to rcceph script

 - only start OSDs if mon daemons are also present
 - adds support for mask and unmask
 - removes support for cluster with non default cluster name,
   as this was very limited and inconsistent
 - Reapplied from a patch as could not cherry-pick
   66cb46c411d874be009c225450eea5021cf1219b from Mon Jan 12
   as this produced issues with src/gmock

Signed-off-by: Owen Synge <osynge@suse.com>

osdc: refactor use of _is_readable

Limit calls to _is_readable to places where the
readableness can actually have changed (on assimilate_prefetch
where it can go true, and in try_read_entry where it
can go false).  Elsewhere, return a cached
result (Journaler::readable).

Signed-off-by: John Spray <john.spray@redhat.com>

osdc: handle corruption in journal

Previously we just *detected* it in _is_readable
and raise a buffer::error.  Now actually catch that
exception and pass up to outer error handling.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: additional error error handling in CDir

Including corruption in fnode+dentries

Signed-off-by: John Spray <john.spray@redhat.com>

mds: refactor BADFRAG setting

So that the procedure around setting STATE_BADFRAG
on a CDir can be used from other forthcoming error
handling.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: refactor CDir::_omap_fetched

Move the dentry decoding into _load_dentry
so that it can subsequently be wrapped
with error handling.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: don't recreate damaged stray dirfrags

Instead of recreating them, go damaged.  Be warned,
the codepath for recreating them by hand doesn't
exist yet (cephfs-data-scan doesn't touch stray
dirs)

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add MDS::damaged_unlocked helper

...to avoid doing a lock/damaged/unlock in lots
of places in MDLog.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: fix MDLog error handling on ENOENT

This was trying to check for standby replay
by testing is_readonly(), but that's always
true during normal replay.  Check mds->is_standby_replay()
instead.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: fix a damaged() case

Neglected to take lock around this call.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: fix corrupt CInode handling

Were previously catching errors from InodeStore but
not the leading magic.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: fix DAMAGED beacons from rejoin

In certain cases, the beacon was emitted from within
handle_mds_map (via MDCache::rejoin_start and
the "No subtrees found for root MDS rank!" error
handler.  This was happening before Beacon::notify_mdsmap
had been called with the new epoch, so the DAMAGED
beacon had the old epoch set, and was consequently
ignored by the MDSMonitor.

The symptom would be that MDS ranks with metadata damaged
in this particular way would flap instead of gracefully
going damaged.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: handle missing stray dirfrags

Fixes case where if dirfrag object missing, ::fetch
would leave in a complete+badfrag state, and
subsequent retryopenroot would hit !is_complete()
assertion on fetch.

Signed-off-by: John Spray <john.spray@redhat.com>

include/buffer: fix constness of operator<< for error

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #4931 from yuyuyu101/wip-perf-msgr

PerfMsgr: A tool to benchmark messenger module

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4994 from liewegas/wip-mon-fixes

mon: fix warning, and get_bucket_utilization

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3566 from majianpeng/librbd-cache-iohint

Librbd cache iohint

Reviewed-by: Josh Durgin <jdurgin@redhat.com>