mds: mark_down old sessions using con

Use the Connection ref to mark old client sessions down.  This is cleaner.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge branch 'wip-4390-b'

Reviewed-by: Greg Farnum <greg@inktank.com>

client: fix kick_flushing_caps() args

Signed-off-by: Sage Weil <sage@inktank.com>

client: clean up useless MetaSession lookups

Signed-off-by: Sage Weil <sage@inktank.com>

client: fix trim_caps() args

Signed-off-by: Sage Weil <sage@inktank.com>

client: fix wake_inode_waiters() arg

Signed-off-by: Sage Weil <sage@inktank.com>

client: validate/lookup mds session in each message handler

For every message handler, look up the MetaSession by int mds and verify
that the Connection* matches properly.  If so, proceed; otherwise, discard
the message.

In the future, we probably want to link the MetaSession to the Connection's
priv field, but that can come later.

Clean up a bunch of submethods that take int mds while we're here.

Signed-off-by: Sage Weil <sage@inktank.com>

client: fix remaining s->inst users to use s->con

Signed-off-by: Sage Weil <sage@inktank.com>

client: remove unused MetaSession::was_stale

Signed-off-by: Sage Weil <sage@inktank.com>

client: only signal session waiters on open and close

Signed-off-by: Sage Weil <sage@inktank.com>

client: use Connection* in MetaSession instead of the inst/addr

This moves us to a cleaner API, and makes the session lifetime explicit
and clean.  Yay!

Signed-off-by: Sage Weil <sage@inktank.com>

client: use session->inst where possible

Removing remaining lookups up the mds inst in the MDSMap.  Instead, use
the MetaSession inst field.

Signed-off-by: Sage Weil <sage@inktank.com>

client: drop unused release_lease()

Signed-off-by: Sage Weil <sage@inktank.com>

client: pass session to send_reconnect and resend_unsafe_requests

Signed-off-by: Sage Weil <sage@inktank.com>

client: switch kick_requests and send_request to take a session *

More cleanup!

Signed-off-by: Sage Weil <sage@inktank.com>

client: handle ESTALE redirection in make_request(), not reply handler

Resending the request in the reply handler is a bit fugly and throws a
small wrench into moving to a MetaSession*-based approach.  Check for
the case(s) where we *do* return ESTALE explicitly and fall through.
Otherwise, kick the caller and let them retry.

Signed-off-by: Sage Weil <sage@inktank.com>

client: fix up _closed_mds_session

Drop useless arg.  Simplify callers.

Signed-off-by: Sage Weil <sage@inktank.com>

client: use helper for _close_mds_session()

Signed-off-by: Sage Weil <sage@inktank.com>

mds: flush log when we hit xlock during path traversal

This ensures we don't stall out waiting for a lock state to change.

This fixes ~4-5 second stalls easily reproducible and visible with
ceph-fuse and 'dbench 1'.

Signed-off-by: Sage Weil <sage@inktank.com>

client: instantiate MetaSession for opening connections

The previous kludge where a waiting_for_session key indicated that we
had an open in progress was... kludgey.

Introduce some helpers to do the session creation/open.

Move the waiting list to be a session member, and clean up associated
code.

Signed-off-by: Sage Weil <sage@inktank.com>

client: pass around MetaSession* instead of int mds

This is mostly just shuffling argument types around.  In a few cases we
now assert that the session actually exists; these would have also been
problematic before when we call get_inst() on bad addrs or something, or
silently ignored bugs.

Signed-off-by: Sage Weil <sage@inktank.com>

client: check for mds state before sending messages

This fixes some callers, not all.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: increase max_global_id before we hit the wall

Bump the max before we run out of IDs to allocate.  This avoids a stall in
authentication every N new clients.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Luis <joao.luis@inktank.com>

Merge branch 'next'

mon: simplify assign_global_id()

Simplify the logic a bit so it is easier to follow.

Small behavior change: we will successfully allocate and return a gid that
== the max when we can't bump it.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: AuthMonitor: don't return global_id right away if we're increasing it

This only happens on the Leader and leads to duplicate global_ids.

Fixes: #4285
Signed-off-by: Joao Luis <joao.luis@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

mon: Paxos: only finish a queued proposal if there's actually *any*

When proposing an older value learned during recovery, we don't create
a queued proposal -- we go straight through Paxos.  Therefore, when
finishing a proposal, we must be sure that we have a proposal in the queue
before dereferencing it, otherwise we will segfault.

Fixes: #4250
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

auth: assert if auth_debug = true and secret_id == 0

Hunting #4282.

Signed-off-by: Sage Weil <sage@inktank.com>

doc: Added libvirt config comments for discard granularity.

fixes: #4395

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

rgw: set up curl with CURL_NOSIGNAL

Fixes: #4425
Backport: bobtail
Apparently, libcurl needs that in order to be thread safe. Side
effect is that if libcurl is not compiled with c-ares support,
domain name lookups are not going to time out.
Issue affected keystone.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

doc: Fixed syntax error.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added tcp and bind settings and cleaned up syntax a bit.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Setting should use network not single IP.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Fixed usage and added note for copying keyring to client. Added hyperlinks.

fixes: #4373

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Addressed confusion with version numbering.

fixes: #4411

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added troubleshooting entry for single OSD and single replica.

fixes: #3327

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge remote-tracking branch 'origin/wip-fix-librados-headers'

Merge pull request #97 from Elbandi/master

Fix FTBFS

Reviewed-by: Gary Lowell <gary.lowell@inktank.com>

Fix typo in ceph usage output

Signed-off-by: David Zafman <david.zafman@inktank.com>

Merge pull request #103 from dalgaaf/wip-da-fix-c_str

mon/Monitor.h: return string instead of 'char *' from get_state_name()

Reviewed-by: Sage Weil <sage@inktank.com>

mon/Monitor.h: return string instead of 'char *' from get_state_name()

Return a string instead of 'char *' to avoid usage of std::string:c_str()
to return a 'char *' from get_state_name().

Returning result of c_str() from a function is dangerous since the
result gets (may) invalid after the related string object gets
destroyed or out of scope (which is the case with return). So you may
end up with garbage in this case.

Related warning from cppcheck:
 [src/mon/Monitor.h:172]: (error) Dangerous usage of c_str(). The value
 returned by c_str() is invalid after this call.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #102 from dalgaaf/wip-da-sca-style-4-v2

Fixes for more cppcheck issues (v2 of #101)

Reviewed-by: Sage Weil <sage@inktank.com>

client/hadoop/CephFSInterface.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

cls/rgw/cls_rgw.cc: prefer prefix --operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

client/Inode.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

SyntheticClient.h: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

CephxKeyServer.h: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

auth/Auth.h: prefer prefix ++operator for non-primitive types

Prefer prefix ++operator for non-primitive types like iterators for
performance reasons. Prefix ++/-- operators avoid creating a temporary
copy.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

tools/rest_bench.cc: remove unused variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/test_filejournal.cc: use %u to format unsigned in sprintf()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/system/st_rados_list_objects.cc: remove not needed statement

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/osd/RadosModel.h: remove unused variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/osd/Object.cc: catch exception by reference

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/omap_bench.cc: remove unused variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test_mon_workloadgen.cc: remove unused variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/librados/aio.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/encoding/ceph_dencoder.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

bench/dumb_backend.h: remove never executed delete statement

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test_object_map.cc: use %u to format unsigned in snprintf()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test_keyvaluedb_iterators.cc: remove unused variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_xml.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_user.cc: remove unused variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_rest_usage.cc: remove unused variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_rest_s3.cc: remove unused variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_rados.cc: perform initialization of prefix in initialization list

Perform initialization of IntentLogNameFilter:prefix in initialization list
instead of constructor body.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_rados.cc: remove unused variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_rados.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

testing: update wordcount to new code layout.

Signed-off-by: Joe Buck <jbbuck@gmail.com>

testing: fix hadoop-internal-tests

The hadoop-internal-tests workunit needs
to be updated in light of our moving to
using stock Hadoop with our hadoop-cephfs
jars.

Signed-off-by: Joe Buck <jbbuck@gmail.com>

ceph-disk-activate: rely on default/configured keyring path

No reason to override the default or configured value here.

Signed-off-by: Sage Weil <sage@inktank.com>

ceph-disk-activate: don't override default or configured osd journal path

There is no reason not to rely on the default or obey any configured
value here.

Fixes: #4031
Signed-off-by: Sage Weil <sage@inktank.com>

Merge branch 'next'

Merge branch 'wip-rbd-flatten-cache' into next

Reviewed-by: Sage Weil <sage.weil@inktank.com>

debian: stop ceph-mds before uninstalling ceph-mds

Fixes: #4384
Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/next'

log: drop default 'log max recent' from 100k -> 10k

Use less memory.

Signed-off-by: Sage Weil <sage@inktank.com>

librbd: invalidate cache when flattening

The cache stores which objects don't exist. Flatten bypasses the cache
when doing its copyups, so when it is done the -ENOENT from the cache
is treated as zeroes instead of 'need to read from parent'.

Clients that have the image open need to forgot about the cached
non-existent objects as well. Do this during ictx_refresh, while the
parent_lock is held exclusively so no new reads from the parent can
happen until the updated parent metadata is visible, so no new reads
from the parent will occur.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

ObjectCacher: add a method to clear -ENOENT caching

Clear the exists and complete flags for any objects that have exists
set to false, and force any in-flight reads to retry if they get
-ENOENT instead of generating zeros.

This is useful for getting the cache into a consistent state for rbd
after an image has been flattened, since many objects which previously
did not exist and went up to the parent to retrieve data may now exist
in the child.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

ObjectCacher: keep track of outstanding reads on an object

Reads always use C_ReadFinish as a callback (and they are the only
user of this callback). Keep an xlist of these for each object, so
they can remove themselves as they finish. To prevent racing requests
and with discard removing objects from the cache, clear the xlist in
the object destructor, so if the Object is still valid the set_item
will still be on the list.

Make the ObjectCacher constructor take an Object* instead of the pool
and object id, which are derived from the Object* anyway.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

ceph: fix cli test

Signed-off-by: Sage Weil <sage@inktank.com>

Revert "Use start-stop-daemon --chuid option to setuid to www-data."

This reverts commit a99ed038ecaf170574ee75cd5ddb12f252f9bc26.

On second thought, this will require a bit more care to ensure that all
of the paths radosgw needs to read/write from have the correct permissions
in the packages and so forth.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge branch 'wip-4135'

Feature: #4135 - osd: noscrub, nodeepscrub osdmap flags

Reviewed-by: Samuel Just <sam.just@inktank.com>

mon: Paxos: increase trim tolerance from 3 to 30.

This increase only means that we'll keep more versions around before we
trim.  It doesn't change the number of versions we'll keep around after
trimming (that's still as much as 'paxos_max_join_drift', i.e. 10), nor
does it change the criteria used to consider a monitor as having drifted
(same rule applies, 'paxos_max_join_drift').

This change however will enable the leader to put off trimming for a longer
period of time, giving a better chance for a monitor to join the cluster.
See, after going through the probing phase, at which point a monitor may
only be, say, 5 versions off, the same monitor may end up getting into the
quorum only to find that in-between probing and finally triggering an
election some 6 versions might have come to existence.  Before this patch,
by then the state had been trimmed and the monitor would have to bootstrap
to perform a full store sync.  With this patch in place, the monitor would
be able to sync the remaining 11 versions.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

mon: Paxos: bootstrap leader if he has fallen behind upon reaching collect

Fixes: #4256
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

Document scrubbing behavior

Signed-off-by: David Zafman <david.zafman@inktank.com>

Merge pull request #100 from jaharkes/init

Fixes for RadosGW init script

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #99 from dalgaaf/wip-da-sca-style-3

More smaller (style related) fixes

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #91 from Youscribe/defaultweight

Set weight of an osd based on df.

Reviewed-by: Sage Weil <sage@inktank.com>

Use start-stop-daemon --chuid option to setuid to www-data.

Signed-off-by: Jan Harkes <jaharkes@cs.cmu.edu>

Fix radosgw actually reloading after rotating logs.

The --signal argument to Debian's start-stop-daemon doesn't
make it send a signal, but defines which signal should be send
when --stop is specified.

Signed-off-by: Jan Harkes <jaharkes@cs.cmu.edu>

rgw/rgw_op.cc: remove unused variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_op.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_multi_del.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_multi.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_http_client.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados_sync.cc: fix typo in error msg

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_common.cc: initialize has_acl_header in constructor with 'false'

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_admin.cc: remove some unused variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_admin.cc: reduce scope of 'int ret'

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_admin.cc: remove twice check for 'object'

Remove twice checked 'object'. Put checks in alphabetic order to make
double checks easier to catch.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>