mds: remove unused MMDSCacheRejoin::{MISSING,FULL}

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: properly wake up dentry waiters after fragmenting dirfrag

When active MDS wants to fragment a replica dirfrag, it should set
the 'replay' parameter of MDCache::adjust_dir_fragments() to false.
It makes sure that CDir::merge/split wake up any dentry waiter.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: choose MIX state if replica of scatterlock is in MIX state

After ScatterLock::infer_state_from_strong_rejoin() set scatterlock
to LOCK_MIX state, don't change the scatterlock to other state.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: switch flushing ScatterLock to dirty ScatterLock after cache rejoin

Otherwise the flushing flag may confuse Locker::eval_gather() if MDS later
imports lock's parent inode.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: handle export freezen race

handle following sequence of events:
- subtree becomes frozen, C_MDC_ExportFreeze is queued.
- export is cancelled
- C_MDC_ExportFreeze is executed.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: allow negetive rstat

When splitting dirfrag, delta rstat is always added to the first new
dirfrag. Ancestors of the dirfrag may have nagtive rstat before the
delta rstat has been propagated to them. For example:

inode 100 n(v1 b-4096)
|- dir 100 n(v1 b-4096)
   |- dentry
      |- inode 101 n(v1 b-4096)
         |- dir 101.0* n(v1)/n(v1 b-4096)
         |- dir 101.1* n(v1)

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: cancel fragmenting dirfrags when cluster is degraded

when cluster is degraded, acquiring locking can take long time.
It is not good to keep dirfrags in frozen state for a long time.

To allow cancelling fragment operation at any time, this commit
also reorganizes the codes that freeze/fetch dirfrags.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: fix _rollback_repair_dir()

_rollback_repair_dir() may increase dirfrag's rfiles/rsubdirs twice.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: fix root and mdsdir inodes' rsubdirs

inode rstat accounts inode itself.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: ignore stale rstat/fragstat when splitting dirfrag

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: encode dirfrag base in cache rejoin ack

Makes sure recovering MDS get uptodate fragstat/rstat for subtree root
dirfrags. it's required the codes that fix-up inode's fragstat/rstat.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: fix-up inode's fragstat/rstat according its dirfrags

Extend the code that fixup inode's fragstat/rstat to handle multiple
dirfrags

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: clear aborted flag before rollback slave requests

There is a special case that the MDRequest needs to be preserved after
rolling back slave rename. The preserved MDRequest will be used later.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: remove mdsdir in the final step of shutdown MDS

Otherwise we may get bad subtree map if we restart the MDS before
the shutdown process finishes.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: tolerate bad sessionmap during journal replay

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: pre-allocate inode numbers less frequently

no need to refill the pre-allocated inode numbers each time an inode
number is used.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: fix frozen inode check in MDCache::handle_discover()

When MDCache::handle_discover() encounters a frozen dirfrag, it should
proceed if the dirfrag is being merged, but the MDS hasn't frozen all
dirfrags yet. When MDCache::handle_discover() checks if a inode is
frozen, it should use CInode::is_frozen_inode() (which doesn't check if
inode's parent dirfrag is frozen).

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: include authpinned objects in remote authpin request

Server::handle_slave_auth_pin() may drop old authpins if it encounters
object that is not authpinable. So it is better to include objects that
have already been authpinned in the remote authpin reuqest.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: send dentry unlink message to replicas of stray dentry

stray dentry may have more replicas than the unlinked dentry has.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: maintain auth bits during replay

Objects' STATE_AUTH bits are set when replaying EImportStart event.
MDCache::trim_non_auth_subtree() clear objects' STATE_AUTH bits when
replaying EExport event.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

Merge pull request #1774 from ceph/wip-8296

osd/ReplicatedPG: fix whiteouts for other cache mode

Reviewed-by: Samuel Just <sam.just@inktank.com>

osd/ReplicatedPG: fix whiteouts for other cache mode

We were special casing WRITEBACK mode for handling whiteouts; this needs to
also include the FORWARD and READONLY modes.  To avoid having to list
specific cache modes, though, just check != NONE.

Fixes: #8296
Backport: firefly
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1601 from ceph/wip-7576

osd: prevent pg map epochs from lagging too far behind

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1748 from onlyjob/docs

sample.ceph.conf update:

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1653 from ceph/wip-7499

rgw, radosgw-admin: bucket link uses bucket instance id now

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1768 from daniel-j-h/code_quality

Variable length array of std::strings (not legal in C++) changed to std::vector<std::string>

Reviewed-by: Sage Weil <sage@inktank.com>

Revert "Fix installation into user home directory, broken by d3f0c0b"

This reverts commit 7539281037ce7a755ac8661ecb15aea32e5f79f6.

This breaks mount.fuse.ceph installation.

doc: Fixed artifacts from merge.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added sudo to setenforce. Restored merge artifact.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added erasure coding and cache tiering notes. Special thanks to Loic Dachary.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Variable length array of std::strings (not legal in C++) changed to std::vector<std::string>

Signed-off-by: Daniel J. Hofmann <daniel@trvx.org>

Merge pull request #1770 from ceph/wip-8290

client: check snap_caps in Incode::is_any_caps()

Reviewed-by: Sage Weil <sage@inktank.com>

client: check snap_caps in Incode::is_any_caps()

Fixes: #8290
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

Merge pull request #1764 from eile/master

Fix installation into user home directory, broken by d3f0c0b

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1769 from ceph/wip-doc-cache-tier

doc: Support for cache tiering.

Merge remote-tracking branch 'origin/wip-doc-radosgw-80'

Conflicts:
doc/radosgw/config.rst

Merge pull request #1760 from ceph/wip-8283

osd/ReplicatedPG: fix trim of in-flight hit_sets

Reviewed-by: Samuel Just <sam.just@inktank.com>

doc: Fix hyperlink.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Index update and librados.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Quotas for Admin Ops API.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: New Admin Guide for Ceph Object Storage.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Fix installation into user home directory, broken by d3f0c0b

Signed-off-by: Stefan Eilemann <Stefan.Eilemann@epfl.ch>

osd: check blacklisted clients in ReplicatedPG::do_op()

OSD checks if client is blacklisted only when receiving OSD request.
It's possible that OSD request's sender get blacklisted while OSD
request in in some waiting list.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
(cherry picked from commit f92677c5b2cac03f42f15c826bfb809fec27830d)

Merge pull request #1740 from ceph/wip-8155

mon: OSDMonitor: disallow nonsensical cache-mode transitions

Reviewed-by: Sage Weil <sage@inktank.com>

ceph-object-corpus: rebase onto firefly corpus

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1762 from yuyuyu101/wip-8282

Fix clone problem

Backport: firefly
Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1752 from ceph/wip-da-SCA-fixes-20140501

Various fixes from SCA

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1755 from eile/master

Fix out of source builds

Reviewed-by: Sage Weil <sage@inktank.com>

Fix out of source builds

Signed-off-by: Stefan Eilemann <Stefan.Eilemann@epfl.ch>

Fix clone problem

When clone happened, the origin header also will be updated in GenericObjectMap,
so the new header wraper(StripObjectHeader) should be updated too.

Fix #8282
Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

mon: OSDMonitor: disallow nonsensical cache-mode transitions

Fixes: 8155
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #1735 from ceph/wip-8113

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1698 from ceph/wip-snapmapper-debug

osd/SnapMapper: debug

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1694 from ceph/wip-throttle-snap-master

osd: throttle snap trimmming with simple delay

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1759 from dachary/wip-mailmap

DNM: mailmap updates

Reviewed-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/ReplicatedPG: fix trim of in-flight hit_sets

We normally need to stat the hit_set to know how many bytes to adjust the
stats by.  If the hit_set was just written, we will get ENOENT.

Get the obc instead, which will either get the in-memory copy (because the
repop is still in flight) or load it off of disk.

Fixes: #8283
Backport: firefly
Signed-off-by: Sage Weil <sage@inktank.com>

Revert "ReplicatedPG: block scrub on blocked object contexts"

This reverts commit e66f2e36c06ca00c1147f922d3513f56b122a5c0.
Reviewed-by: Sage Weil <sage@inktank.com>

osd: Prevent divide by zero in agent_choose_mode()

Fixes: #8175
Backport: firefly

Signed-off-by: David Zafman <david.zafman@inktank.com>
Signed-off-by: Sage Weil <sage@inktank.com>

osd, common: If agent_work() finds no objs to work on delay 5 (default) secs

Add config osd_agent_delay_time of 5 seconds
Honor delay by ignoring agent_choose_mode() calls
Add tier_delay to logger
Treat restart after delay like we were previously idle

Fixes: #8113
Backport: firefly

Signed-off-by: David Zafman <david.zafman@inktank.com>

rados.cc: fix typo in help output

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Dumper::dump_entries(): reduce scope of 'got_data'

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

PG::read_info(): pass 'const coll_t coll' by reference

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

PGBackend::be_compare_scrubmaps(): pass pgid by reference

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd_types.h: pass eversion_t by reference to operator<<

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

SimpleLock.h: remove unused private function clear_more()

Remove unused private function clear_more(), it's replaced by
try_clear_more().

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

SimpleLock.h: remove twice included osd_types.h

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

linux_fiemap.h: remove twice included int_types.h

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Dumper::dump_entries(): remove not needed variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw_bucket.cc: return error if update_containers_stats() fails

In case need_stats is set on rgw_read_user_buckets() and the
update_containers_stats() call fails with !-ENOENT, not only
print out a message but also return the error to the function
caller.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/PGMonitor: set tid on no-op PGStatsAck

The OSD needs to know the tid.  Both generally, and specifically because
the flush_pg_stats may be blocking on it.

Fixes: #8280
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon/OSDMonitor: share latest map with osd on dup boot message

If we get a dup boot message, share the newer maps with the osd so that
they know they are living in the past.

Fixes: #8279
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon/MonClient: remove stray _finish_hunting() calls

Callig _finish_hunting() clears out the bool hunting flag, which means we
don't retry by connection to another mon periodically.  Instead, we send
keepalives every 10s.  But, since we aren't yet in state HAVE_SESSION, we
don't check that the keepalives are getting responses.  This means that an
ill-timed connection reset (say, after we get a MonMap, but before we
finish authenticating) can drop the monc into a black hole that does not
retry.

Instead, we should *only* call _finish_hunting() when we complete the
authentication handshake.

Fixes: #8278
Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

mailmap: Florent Bautista affiliation

and name normalization

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Warren Usui name normalization

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Guang Yang name normalization

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1754 from nereocystis/hardware-to-glossary

doc: Include links from hardware-recommendations to glossary

sample.ceph.conf update:

 * corrected URLs.
 * added [client] section.
 * more options and descriptions.
 * filestore settings were moved under [osd].
 * cephx settings to reflect lack of support for authentication in RBD client.
 * many minor corrections and updates.

Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>

osd/ReplicatedPG: agent_work() fix next if finished early due to start_max

Backport: firefly

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd/SnapMapper: pass snaps set by const ref

Signed-off-by: Sage Weil <sage@inktank.com>

osd/SnapMapper: debug

Signed-off-by: Sage Weil <sage@inktank.com>

mon/OSDMonitor: do not reply to MOSDMarkMeDown if ack is not requested

If a reply isn't requested, do not bother to send one.  Note that old
clients did not request an ack, but we will infer that they do from the
message encoding (see previous commit).

Signed-off-by: Sage Weil <sage@inktank.com>

osd: fix 'ack' to be 'request_ack' in MOSDMarkMeDown

This field was passed along but always set to false.  It did not seem to
indicate whether this was/wasn't an ack (the reply message kept the same
value) or whether an ack was requested (it was always false but we always
reply).

Change this by bumping the message encoding, renaming the field, and
making old messages that are decoded force request_ack to true.  Reply
messages will always have request_ack false (although that part doesn't
matter much since the fact that it is a reply is clear from context).

Signed-off-by: Sage Weil <sage@inktank.com>

osd: ignore MarkMeDown message if we aren't in PREPARING_TO_STOP state

If we aren't waiting for this, ignore it.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: prevent pgs from getting too far ahead of the min pg epoch

Bound the range of PG epochs between the slowest and fastest pg
(epoch-wise) with 'osd map max advance'.  This value should be set to
something less than 'osd map cache size' so that the maps we are
processing will be in memory as many PGs advance forward in time in
loose synchrony.

This is part of the solution to #7576.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: track per-pg epochs, min

Add some simple tracking so that we can quickly determine what the min
pg osdmap epoch is.

Signed-off-by: Sage Weil <sage@inktank.com>

doc: Include links from hardware-recommendations to glossary
Included :term: in parts of hardware-recommendations so that glossary
links appear.
Signed-off-by: Kevin Dalley <kevin@kelphead.org>

Merge pull request #1749 from hufman/fix-typo-releasenotes-pyramind

Fixes a very minor typo in the release notes

Reviewed-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds: note MDiscoverReply encoding change in corpus

Signed-off-by: Sage Weil <sage@inktank.com>

mds: bump protocol

In commit f689e5f049736bb0a0fa437e05936f6c1b9c1bb6 we change the encoding
and semantics for MDiscoverReply.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1738 from ceph/wip-8147

osd: automatically scrub PGs with invalid stats

Reviewed-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: David Zafman <david.zafman@inktank.com>

doc/release-notes: changelog link

Signed-off-by: Sage Weil <sage@inktank.com>

doc/release-notes: final v0.67.8 notes and changelog

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1750 from nereocystis/doc-link-to-involved

doc: documenting links to get-involved

Fixes a very minor typo in the release notes

pyramind -> pyramid

doc: documenting links to get-involved
Create a link from documenting-ceph so that it is easy to find the
github repository used for ceph.

Signed-off-by: Kevin Dalley <kevin@kelphead.org>

ReplicatedPG: block scrub on blocked object contexts

Fixes: #8011
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1745 from ceph/wip-7941

rados.h,ReplicatedPG: add CEPH_OSD_FLAG_ENFORCE_SNAPC and use on flush

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1744 from ceph/wip-8161

ECBackend::continue_recovery_op: handle a source shard going down

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1746 from ceph/wip-8086

Wip 8086

Reviewed-by: Sage Weil <sage@inktank.com>

rgw_common.cc: reduce scope of 'fpos' variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>