doc: Provided additional detail on daemon-specific caps syntax.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Updated Keyring settings from comments and ceph-deploy defaults.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added links. Clarified namespace comments. Added limitations.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Minor changes.

Removed arch-related comments.
Added links to relevant docs.
Moved limitations to user management.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added a few comments and links to other relevant docs.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added user management link to quick start.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Removed cephx intro. Moved details to user management, config, and architecture.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Removed cephx guide. Replaced by user management.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Removed auth intro and auth docs and added user management to index.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Restructured auth configuration reference.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Put architectural details of authentication in to architecture doc.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Created a new User Management doc to replace authentication.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge remote-tracking branch 'upstream/next'

Merge pull request #2252 from ceph/wip-9087

test/system/systest_runnable.cc: debugging on start and end

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

test/system/systest_runnable.cc: debugging on start and end

Signed-off-by: Samuel Just <sam.just@inktank.com>

mon: fix divide by zero when pg_num adjusted and no osds

Fixes: #9052
Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@redhat.com>

mon: fix potential divide by zero on can_mark_{down,out}

Signed-off-by: Sage Weil <sage@redhat.com>

mon: fix divide by zero when pg_num adjusted and no osds

Fixes: #9101
Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@redhat.com>

mon: fix potential divide by zero on can_mark_{down,out}

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2236 from ceph/wip-9055

ceph_test_rados_api_tier: fix cache pool cleanup during test

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2222 from ceph/wip-9029

mds: Make min/max UID configurable for who is allowed to create a snapsh...

Reviewed-by: Sage Weil <sage@redhat.com>

mds: Revert from mds_mksnap_ setting to mds_snap_ settings

Merge pull request #2249 from ceph/wip-9096

osd: fix require_same_peer_instance from fast_dispatch

Reviewed-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

osd/ReplicatedPG: only do agent mode calculations for positive values

After a split we can get negative values here.  Only do the arithmetic if
we have a valid (positive) value that won't through the floating point
unit for a loop.

Fixes: #9082
Tested-by: Karan Singh <karan.singh@csc.fi>
Signed-off-by: Sage Weil <sage@redhat.com>

osd: fix some line wrapping

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fix require_same_peer_instance from fast_dispatch

The mark-down of old peers needs to take the session_dispatch_lock in order
to safely clear the Session ref cycle.  However, for fast dispatch callers,
that lock is already held.  Pass a flag down from the callers indicating
whether we need to take the additional lock.

Fixes: #9096
Signed-off-by: Sage Weil <sage@redhat.com>

osd: inline require_osd_up_peer

There is only one caller.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2233 from majianpeng/fix1

os/chain_xattr: Remove all old xattr entry when overwrite the xattr.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2230 from ceph/wip-fsx-flatten

test_librbd_fsx: also flatten as part of randomize_parent_overlap

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2234 from kevincox/net-docs

doc: Initial network docs.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2237 from ceph/wip-8560

mon: instrument paxos

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon/Paxos: add perfcounters for most paxos operations

I'm focusing primarily on the ones that result in IO here.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2242 from majianpeng/fix4

utime: Because class Clock didn't exist,so remove the declaration in class utime_t

Reviewed-by: Sage Weil <sage@redhat.com>

Revert "ReplicatedPG: do not pass cop into C_Copyfrom"

The ref was introduced in 589b639af7c8834a1e6293d58d77a9c440107bc3
and is actually necessary to keep the buffers around.

This reverts commit 300b5e8ecbb7526b55e2cb5eeba81fd501a8b652.

ReplicatedPG: do not pass cop into C_Copyfrom

We do not know when the objecter will finally let go of this Context.  Thus, we
cannot know whether it will happen before the flush, at which point the
object_context held by the cop must have been released.

Also, we simply don't need it, process_copy_chunk alrady works in terms of the
tid!

Fixes: #8894
Reviewed-by: Sage Weil <sage@redhat.com>
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2246 from ceph/wip-9064

ReplicatedPG::maybe_handle_cache: do not skip promote for write_ordered

Reviewed-by: Sage Weil <sage@redhat.com>

ReplicatedPG::maybe_handle_cache: do not skip promote for write_ordered

We cannot redirect a RW ordered read.

Fixes: #9064
Introduced: 0ed3adc1e0a74bf9548d1d956aece11f019afee0
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2245 from dachary/wip-9085-isa-link

erasure-code: isa plugin must link with ErasureCode.cc

Reviewed-by: Sage Weil <sage@redhat.com>

erasure-code: isa plugin must link with ErasureCode.cc

Otherwise it will not get the methods it needs. A test is added to check
the plugin loads as expected, from the command line. The test is not run
if the isa plugin is not found, which happens on platforms that are not
supported.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #2239 from ceph/wip-8912

librbd: fix error path cleanup for opening an image

Reviewed-by: Sage Weil <sage@redhat.com>

doc/changelog: v0.67.10 notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-8860'

Reviewed-by: Sage Weil <sage@redhat.com>

init-ceph: conditionally update  after argparsing

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

doc/release-notes: v0.67.10

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

Conflicts:
PendingReleaseNotes

Merge pull request #2243 from ceph/wip-log-client

common/LogClient: fix sending dup log items

Reviewed-by: Loic Dachary <loic@dachary.org>

common/LogClient: fix sending dup log items

We need to skip even the most recently sent item in order to get to the
ones we haven't sent yet.

Fixes: #9080
Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2232 from dachary/wip-9044-use-ruleset

erasure-code: ErasureCodeJerasure::create_ruleset must return a ruleset

Reviewed-by: Xiaoxi Chen <xiaoxi.chen@intel.com>
Reviewed-by: Ma Jianpeng <jianpeng.ma@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

utime: Because class Clock didn't exist,so remove the declaration in class utime_t.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>

doc: Replace [default] with [global].

Fixes: #8955
Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added yum-priorities procedure to manual install.

Fixes: #8734
Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added priority = 2 to packages.

Fixes: #8734
Signed-off-by: John Wilkins <john.wilkins@inktank.com>

librbd: fix error path cleanup for opening an image

If the image doesn't exist and caching is enabled, the ObjectCacher
was not being shutdown, and the ImageCtx was leaked. The IoCtx could
later be closed while the ObjectCacher was still running, resulting in
a segfault. Simply use the usual cleanup path in open_image(), which
works fine here.

Fixes: #8912
Backport: dumpling, firefly
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

rgw: Don't send error body when it's a HEAD request

The main 'operation' know not to send any body for HEAD requests.

However for errors, this was not the case, the formatter would be
flushed and would send the error 'message' in the body in all cases.

For the FastCGI case it doesn't seem to be an issue, it's possible
that the webserver (apache/lighttpd/...) cleans up the response
into shape.

But when using the new civetweb frontend this cause invalid HTTP.

Backport: firefly
Fixes #8539

Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>
Signed-off-by: Sylvain Munaut <s.munaut@whatever-company.com>

Merge pull request #2214 from ceph/wip-rados-df-json

rados: fix 'rados df --format=json' field names

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

rados: remove {read,write}_kb fields from rados df json output

Redundant.  Do not backport.

Signed-off-by: Sage Weil <sage@redhat.com>

rados: fix {read,write}_bytes value

Fill this in with kb * 1024 instead of the ops count.

Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2238 from ceph/wip-9057

msg/Pipe: do not wait for self in Pipe::stop_and_wait()

Reviewed-by: Samuel Just <sam.just@inktank.com>

msg/Pipe: do not wait for self in Pipe::stop_and_wait()

The fast dispatch code necessitated adding a wait for the fast dispatch
to complete when taking over sockets back in commit
2d5d3097c3998add1061ce253104154d72879237.  This included mark_down()
(although I am not certain mark_down was required to fix the previous set
of races).

In any case, if the fast dispatch thread itself tries to mark down its
own connection, it will deadlock in this method waiting for itself to
return and clear reader_dispatching.  Skip this wait if we are in fact
the reader thread.  This avoids the deadlock.

Alternatively, we could change mark_down() to not use stop_and_wait(), but
I am less clear about the potential races there, so I'm opting for the
minimal (though ugly) fix.

Fixes: #9057
Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonitorDBStore: add get_{keys,bytes}() accounting to Transaction

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_tier: fix cache cleanup (ec too)

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api: fix cleanup of cache pool

We can't simply try to delete everything in there because some items may
be whiteouts.  Instead, flush+evict everything, then remove overlay, and
*then* delete what remains.

Fixes: #9055
Signed-off-by: Sage Weil <sage@redhat.com>

librados/TestCase: inheret cleanup_default_namespace

No need to duplicate this code.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: improve ruleno/ruleset consistency

rename variables to reflect when they hold a ruleset and when they hold
a rule id / ruleno in the hope to avoid bugs in the future.

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: ErasureCodeIsa::create_ruleset must return a ruleset

http://tracker.ceph.com/issues/9044 Refs: #9044

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: ErasureCodeJerasure::create_ruleset must return a ruleset

CrushWrapper::add_simple_ruleset does not return a ruleset, it returns
a ruleid that must be converted into a ruleset before being returned.

http://tracker.ceph.com/issues/9044 Fixes: #9044

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: OSDMonitor::crush_ruleset_create_erasure needs ruleset

When OSDMonitor::crush_ruleset_create_erasure checks the ruleset for
existence, it must convert the ruleid into a ruleset before assigning it
back to the *ruleset parameter.

http://tracker.ceph.com/issues/9044 Fixes: #9044

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #2201 from dachary/wip-8496-erasure-code-base-class

erasure code base class

Reviewed-by: Andreas-Joachim Peters <Andreas.Joachim.Peters@cern.ch>

os/FileStore: dump open fds before asserting

Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@redhat.com>

doc: Initial network docs.

Signed-Off-By: Kevin Cox <kevincox@kevincox.ca>

os/chain_xattr: Remove all old xattr entry when overwrite the xattr.

Ceph use multiple xattrs to store the value of a single xattr which size
is larger than CHAIN_XATTR_MAX_BLOCK_LEN.
But when overwote the content of xattr in func
chain_setxattr/chain_fsetxattr, we don't know the size of previous
content of the xattr.
So we only try to remove until system return  -ENODATA.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>

Merge branch 'next'

Conflicts:
src/osd/OSD.cc

OSD: add require_same_peer_inst(OpRequestRef&,OSDMap&) helper

Signed-off-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit e99acf9810976b1fc74b84ad289773af43be973f)

OSD: introduce require_self_aliveness(OpRequestRef&,epoch_t) function

Take the self-aliveness checks out of require_same_or_newer_map() and use
the new function for that and for require_up_osd_peer().

Signed-off-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit e179e9227b4a4482d8359682092fd7f426b9a919)

OSD: use OpRequestRef& for a few require_* functions

Signed-off-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit eb2f1ea2c33647934af878b504383829f5a198ce)

OSD: introduce require_up_osd_peer() function for gating replica ops

This checks both that a Message originates from an OSD, and that the OSD
is up in the given map epoch.
We use it in handle_replica_op so that we don't inadvertently add operations
from down peers, who might or might not know it.

Signed-off-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit ccd0eec50103b919b3eb6eea96f7dc6438520ed3)

Merge remote-tracking branch 'gh/next'

Merge pull request #2228 from ceph/wip-pg-epoch

osd: fix pg epoch floor tracking

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2209 from somnathr/wip-sd-check-pool-existence

RadosClient: Adding pool existence checks for pool create interfaces.

Reviewed-by: Sage Weil <sage@redhat.com>

RadosClient: Enable pool existence check

Pool create interfaces would now refer to updated osdmap
for pool existence, such that an appropriate message is
thrown to the user when run via CLI.

Fixes: #8835
Signed-off-by: Pavan Rallabhandi <pavan.rallabhandi@sandisk.com>

test_librbd_fsx: also flatten as part of randomize_parent_overlap

With randomize_parent_overlap fsx will randomly truncate base images
after they have been cloned from.  This throws flatten into the mix:
base image will be flattened with 2/16 chance (equal to the chance of
leaving the image intact).

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

mds: Also check min/max uid on snap removal

Not only check if the uid is correct when creating a snapshot, but also
verify if the UID is correct when trying to remove a snapshot.

Fixes: #9029

osd: fix pg epoch floor tracking

If you call erase() on a multiset it will delete all instances of a value;
we only want to delete one of them.  Fix this by passing an iterator.

Backport: firefly
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2226 from athanatos/wip-8396

Wip 8396

Reviewed-by: Sage Weil <sage@redhat.com>

OSD: move waiting_for_pg into the session structures

Each message belongs to a session.  Further, no ordering is implied
between messages which arrived on different sessions.  Breaking the
global waiting_for_pg structure into a per-session structure lets
us avoid the problem of taking a write lock on a global structure
(pg_map_lock) in get_pg_or_queue_for_pg at the cost of some
complexity in updating each session's waiting_for_pg structure when
we receive a new map (due to pg splits) or when we locally create
a pg.

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2227 from athanatos/wip-8625

osd_types: s/stashed/rollback_info_completed and set on create

Reviewed-by: Sage Weil <sage@redhat.com>

doc: Removed Debian reference from Upstart.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Incorporated user feedback to clarify upgrade doc.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added configuration discussion at end of gateway install with links to config.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

OSD::shutdown: actually drop sessions waiting on map

There might be messages for which we still don't have the
map.  Dispatching waiting won't actually help.

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: clear_session_waiting_on_map in ms_handle_reset

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: rename session_waiting_for_map_lock to session_waiting_lock

This lock will also protect the waiting_for_pg structures in each
session.

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: wake_pg_waiters outside of the pgmap write_lock, pg_lock

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: fix wake_pg_waiters revert error in _open_lock_pg

231fe1b685bfbd3db9c81709ca39a29d696b13ad reintroduced erroneously
this call to wake_pg_waiters.  All _create_lock_pg callers handle
calling wake_pg_waiters after the pg lock has been dropped.

Fixes: #8691
Signed-off-by: Samuel Just <sam.just@inktank.com>

osd_types: s/stashed/rollback_info_completed and set on create

Originally, this flag indicated that the object had already been stashed and
that therefore recording subsequent changes is unecessary.  We want to set it
on create() as well since operations like [create, writefull] should not need
to stash the object.

Fixes: #8625
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge remote-tracking branch 'gh/wip-filestore-bigxattr'

make ceph-disk use the new init flag for cluster

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

allow passing a --cluster flag to the init script

It will fallback to looking at /etc/ceph/$cluster.conf
otherwise.

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

mds: Make min/max UID configurable for who is allowed to create a snapshot

This way administrators can configure which UIDs are allowed to create
snapshots on CephFS.

In certain situations it might be desirable that regular users are not allowed
to create snapshots. For example on shared hosting environments.

Fixes: #9029