osd: fix _activate_committed() crash

Do not dereference acting[0] unless we know it is still valid.

Take a reference when scheduling the transaction, and drop it in the
completion, to ensure that the PG isn't removed out from underneath us.

Fixes: #1379
Signed-off-by: Sage Weil <sage@newdream.net>

compatset: dump json

Signed-off-by: Sage Weil <sage@newdream.net>

uninline a bunch of buffer.h stuff

uninline osd types

mds: add debug_mds_locker

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librbd: rbd_list: return buffer size, not element count

Fixes: #1397
Signed-off-by: Sage Weil <sage@newdream.net>

mon: add/remove data_pg_pools from mdsmap

Signed-off-by: Sage Weil <sage@newdream.net>

PG: last_warm_restart must be set to the current epoch

Previously, last_warm_restart did not get set until the
first time the pg was restarted.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

mds: explain weird encoding

Signed-off-by: Sage Weil <sage@newdream.net>

mds: whitespace

Signed-off-by: Sage Weil <sage@newdream.net>

mds: drop unused projected dirfrag_t arg for EMetaBlob::add_primary_dentry

Not used.  If it is later, we need to project it properly.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: drop unused snapbl arg

Signed-off-by: Sage Weil <sage@newdream.net>

mds: avoid explicit passing of projected_xattrs

No need to pass this in explicitly; we can look in the projected inode for
it.  This actually fixes a race where a journaled inode following a
setxattr will not journal the projected xattrs.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: avoid passing snapbl around explicitly

We now properly track projected snaprealm/sr_t's.  Use that for journaling
instead of manually passing around a snapbl.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: use snaprealm projection for MDCache::snaprealm_create()

Signed-off-by: Sage Weil <sage@newdream.net>

mds: simplify sr_t journaling

Drop encode/decode in SnapRealm and encode the sr_t's directly.  This way
the caller says what they mean (current or projected).

Signed-off-by: Sage Weil <sage@newdream.net>

mds: journal inode on mksnap

We were cowing but not journaling a simply-dirty inode.  Der.

Fixes: #1389
Signed-off-by: Sage Weil <sage@newdream.net>

mds: handle O_TRUNC when size is already 0

We always want to go through the truncation path.  Two reasons:

 - even if the size is already 0, we still need to update ctime/mtime
 - we may not have the correct size (client could have dirty data), in
   which case we still want to bump truncate_seq etc.

Basically, we can't trust pi->size here because we are holding a wrlock,
not an xlock.  And that's fine, since we need to update the inode
unconditionally anyway.

This broke cfuse pjd open/00.t tests when we added the fuse option
atomic_o_trunc.  libceph has always been broken in this regard.

Fixes: #1393
Signed-off-by: Sage Weil <sage@newdream.net>

client: send O_TRUNC open requests to MDS

Signed-off-by: Sage Weil <sage@newdream.net>

client: move Inode operator<<

Signed-off-by: Sage Weil <sage@newdream.net>

rbd: remove dup 'rm' command from man page

Fixes: #1395
Signed-off-by: Sage Weil <sage@newdream.net>

cfuse: take out accidental bad channel add

This was a brainstorm that snuck into last commit, grr.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: fix virtual bucket name handling

cfuse: make use of fuse invalidate callback optional (and off by default)

This is hitting a deadlock in fuse itself (or so it appears).  Disable it
until that's sorted out.

Signed-off-by: Sage Weil <sage@newdream.net>

formatter: remove bad asserts

rgw: list objects shows time in GMT, thread safety fix

src/perfglue/heap_profiler.cc: fix snprintf

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

client: compile with older libfuse2

Merge branch 'wip-librados-tests'

osd_types: fix merge

Signed-off-by: Sage Weil <sage@newdream.net>

Merge commit 'v0.33'

v0.33

osd: pg_stat_t: throw exception on unknown encoding

This is better then decoding nonsense, eating ram, etc.

Signed-off-by: Sage Weil <sage@newdream.net>

qa: test rename into snapped dir

Tests fix for #1399.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: mark newly created snaprealms as open

Fixes: #1399
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test/rados-api/misc.cc: test Rados::version

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/list.cc: C++ version

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/snapshots.cc: C++ version

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/watch_notify.cc: implement C++ ver

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/stat.cc: implement C++ tests

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados/librados.h: use rados_snap_t consistently

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/aio.cc: test RoundTripWriteFull

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/aio.cc: implement FlushPP

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/aio.cc: test ReturnValuePP

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/aio.cc: test IsSafePP

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/aio.cc: add IsCompletePP

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/aio.cc: Add RoundTripAppendPP

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/aio.cc: add RoundTripPP

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Fix bug in turning off logging. Remove log_dir.

Fix a bug that made it impossible to turn off logging to a file.

Remove the old "log_dir" setting. It has been deprecated for a long
time, and it adds a lot of complexity to the code.

heap_profiler: use the directory that log_file is in to construct a
profiler output file name. Don't use log_dir.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'next'

osd: fix heartbeats after bad markdown

The heartbeat start message comes from hbin messenger, which has no port
and a nonce of the pid (at startup).  When we mark ourselves down/up, and
then resend a start, the peer will send a RESETSESSION and the stat
message will get lost, and then we'll miss heartbeats.

Mark down all connections, so that when we reconnect, our start message
is not lost.

Signed-off-by: Sage Weil <sage@newdream.net>

monmaptool: don't inc bool

Coverity cid 22

Signed-off-by: Sage Weil <sage@newdream.net>

crushtool: don't inc bool

Coverity cid 21

Signed-off-by: Sage Weil <sage@newdream.net>

mds: break #include recursion

CInode.h -> SessionMap.h -> CInode.h

Coverity cid 14

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: some encoding fixes

rgw: handle zero and invalid max_keys

rgw: create CommonPrefixes section for each prefix

rgw: adjust returned error when got invalid auth request

obsync: add SRC_SECURE, DST_SECURE

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

LibRadosAio: WaitForSafePP

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/aio.cc: SimpleWritePP

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_setxattr: return 0 on success

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_write_full: remove offset parameter

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'next'

librbd: output copy progress

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

mds: avoid issue_caps on snapped inodes

Only head inodes have caps.  Don't set need_issue if it's not a head inode.

This is cleaner than bailing out of issue_caps(); we shouldn't get to that
point. The places that set need_issue = true usually set gather++ too,
which also shouldn't happen on a snapped inode.

Fixes: #1390
Reported-by: Damien Churchill <damoxc@gmail.com>
Signed-off-by: Sage Weil <sage@newdream.net>

test/rados-api/snapshots.cc:RollbackPP

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/snapshots.cc:SnapGetNamePP

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/snapshots.cc:SelfManagedRollbackTest

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

SelfManagedSnapTest: test more

* test more snapshot functions in SelfManagedSnapTest

* rados_ioctx_selfmanaged_snap_set_write_ctx: fix name in C file to
conform to header file

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/snapshots.cc: SelfManagedSnapTestPP

Also test SelfManagedSnapRollbackPP

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/io.cc: test XattrListPP

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/io.cc: test RmXattrPP

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/io.cc: test XattrsRoundTripPP

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

client: use localized ldout macro

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: dump mdsmap in json

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test/rados-api/io.cc: add RemoveTestPP

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/io.cc: AppendRoundTripPP,TruncPP

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/io.cc: test WriteFullRoundTripPP

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/io.cc: test RoundTripPP, etc.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/io.cc: test SimpleWritePP

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rados-api/test.cc: add create_one_pool_pp,etc

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados: add IoCtx::close

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: don't wait for lock 'flushing' flag on replicas

If we are a replica, the 'flushing' means that we had dirty scatterlock
data and are waiting for it to get flushed out to the auth copy (by
cycling from MIX->LOCK, normally).  If we end up with 'flushing' set
while in the MIX state, we can't wait for it to clear before responding
to a lock request from the primary or we'll deadlock.

On the auth, flushing means flushing to the log, which makes sense; that
will always make progress despite scatterlock activity.

This fixes a hang from 3-mds fsstress with thrashing exports.  (Strangely
I never hit this on fatty.)

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: deduplicate sparse read interpretation

AioBlockCompletions and read_iterate each had their own copy of this
code, leading to bugs when only one was changed. Move this to a
separate function, handle_sparse_read.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: make read_iterate mirror AioCompletion::complete

This makes testlibrbd and testlibrbdpp pass again, and
now both I/O paths have the same logic.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

testlibrbd: remove useless print statements

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

testlibrbd: NUL-terminate orig_data

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

testlibrbd(pp): accept standard command line arguments

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: don't request heartbeats from down peers

We try to add sources for anything in peer_info, which can include down
OSDs.  Skip them.

Signed-off-by: Sage Weil <sage@newdream.net>

cfuse: properly invalidate page cache data via callback

The Client has a cache, but it is actually usually duplicating
what the kernel has in the page cache.  Use the Client callback to do a
libfuse upcall to invalidate the kernel page cache when appropriate.

Usually people didn't notice this because we were invalidating the page
cache on every file open. Stop doing that.

This gives us:
 - consistent multi-client semantics for open files
 - fewer read calls because the kernel can (re)use the page cache more
   effectively.

We are still double-caching.  We probably want to be aggressive about not
caching any clean file data in Client from cfuse since the page cache does
a better job than we do.

Signed-off-by: Sage Weil <sage@newdream.net>

client: add callback for cache invalidate

This will get called any time an inode's cached file data is invalidated.
cfuse will use it.

Signed-off-by: Sage Weil <sage@newdream.net>

client: use helper to invalidate/truncate cached file data

We invalidate cached file data in several places; use a common helper.

Signed-off-by: Sage Weil <sage@newdream.net>

client: fix cached file data truncation

We want to truncate our cached data when the file size changes, not
when truncate_size does.  This aligns us with the kernel implementation,
and it works when overwriting a file from multiple clients.

Signed-off-by: Sage Weil <sage@newdream.net>

fuse: enable atomic_o_trunc

The MDS does this atomically.

Signed-off-by: Sage Weil <sage@newdream.net>

fuse: allow big writes

Easiest 500% performance improvement ever!

Signed-off-by: Sage Weil <sage@newdream.net>

Add test/rados-api/misc.cc

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'wip-pg-creation'

librados: implement/document tmap_{get,put}

These aren't strictly necessary now (you can just read the raw object or
do a writefull and get the same thing) but this way we document the format
and can change the backend to be smarter in the future without changing up
the interface.

Signed-off-by: Sage Weil <sage@newdream.net>

test/rados-api/snapshots: add SelfManagedSnapTest

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>