OSD: info.pgid gets removed from creating_pgs before _create_lock_new_pg

Fix bug introduced in cd2716c4151f5c0996bb4d0cf08961f8da281dac.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rbd: --size is megabytes, not bytes

Signed-off-by: Sage Weil <sage@newdream.net>

osd: take ondisk_read_lock on src_oids

We need to take the ondisk read lock on src oids for multiobject operations
(like clonerange) to ensure that written data has hit disk before we
clone it elsewhere.

Order of acquisition doesn't actually matter here, since the ondisk locks
are all leaves in the lock dependency hierarchy.

Signed-off-by: Sage Weil <sage@newdream.net>

object.h: make hobject key private, prevent objects where key == oid

There should be no difference between an object with a key identical to
its name and an object with the same name but no key.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: make loop exit case explicit, removes warning

README: update a bit

Signed-off-by: Sage Weil <sage@newdream.net>

IndexManager: delete index during RemoveOnDelete

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

OSD: use creating_pgs[pgid].history in get_or_create_pg for new pg

If info.pgid is in creating_pgs, we should use the history from
creating_pgs.  The history passed in will be an empty history from a
creation probe in that case.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

client: add inode on IMPORT

If we get an IMPORT and don't have the inode, add it.  This fixes a race
like:

 mds0 -> client .. mknod reply (or similar)
 mds0 -> mds1 .. migrate cap
 mds1 -> client .. send IMPORT
 client <- mds1 .. rx IMPORT, but don't have inode
 client <- mds0 .. rx mknod reply, add it.

With this fix, we add the inode and set up the cap on IMPORT, and when we
get the mknod reply we update the inode immutable fields that aren't
present in the cap message (rdev, symlink target).

Fixes: #1513
Signed-off-by: Sage Weil <sage@newdream.net>

client: factor out flush_cap_releases() helper

Signed-off-by: Sage Weil <sage@newdream.net>

mds: bracket LOCK|AUTH -> PREXLOCK transition with start/finish_locking

Unlike other lock transitions, we are moving to an _unstable_ state and
then taking our (x)lock.  That means that if we don't actually finish for
some reason (lock order changes, request is canceled, whatever) we leave
things in an unstable state--in this case, PREXLOCK, where nothing else
will touch it.

Call cancel_lock in drop_locks (or in acquire_locks when the order changes)
to clean up after an aborted lock attempt.

Original but reproduced (though not easily) by multimds collection task
fsstress_thrash_subtrees.yaml.

Fixes: #1425
Signed-off-by: Sage Weil <sage@newdream.net>

mds: _finish_xlock helper

Factor out _finish_xlock helper to move from XLOCKDONE (or, later,
PREXLOCK) to a non-xlock state.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: mutation: add start/finish locking hooks

Keep track of which lock we are currently working on locking in the
Mutation.  Use helpers to make sure we start/finish locking in a sane
way.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: move Mutation + children out of MDCache.h

Uninline while we're at it.

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: {remove,resize,snap_rollback}_with_progress

C and C++ APIs.  Mirrors the copy_with_progress addition.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: log when we get an error

...before we assert(!err).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph.spec: spec file should be creating /var/run/ceph

Signed-off-by: Brian Chrisman <brchrisman@gmail.com>

osd: nss build of OSD.cc needs CRYPTO_CXXFLAGS since

nss build fails like this:

if /bin/sh ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I. -I.   \
-D__KERNEL_STRICT_NAMES -Wall -D__CEPH__ -D_FILE_OFFSET_BITS=64 -D_REENTRANT \
-D_THREAD_SAFE -D__STDC_FORMAT_MACROS -D_GNU_SOURCE -rdynamic   -Winit-self \
-Wpointer-arith -fno-strict-aliasing -Wnon-virtual-dtor -Wno-invalid-offsetof \
-Wstrict-null-sentinel   -g -MT OSD.lo -MD -MP -MF ".deps/OSD.Tpo" -c -o OSD.lo `test \
                -f 'osd/OSD.cc' || echo './'`osd/OSD.cc; \
        then mv -f ".deps/OSD.Tpo" ".deps/OSD.Plo"; else rm -f ".deps/OSD.Tpo"; exit \
1; fi  g++ -DHAVE_CONFIG_H -I. -I. -I. -D__KERNEL_STRICT_NAMES -Wall -D__CEPH__ \
-D_FILE_OFFSET_BITS=64 -D_REENTRANT -D_THREAD_SAFE -D__STDC_FORMAT_MACROS \
-D_GNU_SOURCE -rdynamic -Winit-self -Wpointer-arith -fno-strict-aliasing \
-Wnon-virtual-dtor -Wno-invalid-offsetof -Wstrict-null-sentinel -g -MT OSD.lo -MD -MP \
-MF .deps/OSD.Tpo -c osd/OSD.cc  -fPIC -DPIC -o .libs/OSD.o In file included from \
./os/LFNIndex.h:27,  from ./os/HashIndex.h:20,
                 from ./os/FileStore.h:26,
                 from osd/OSD.cc:25:
./common/ceph_crypto.h:44:22: error: pk11pub.h: No such file or directory

Commit d59f34ab made FileStore.h include HashIndex.h, which ultimately
results in OSD.cc including ceph_crypto.h.

Signed-off-by: Jim Schutt <jaschut@sandia.gov>

rgw: fix arg parsing

rbd: fix command option parsing, honor --

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cephtool: add --concise option again

This was accidentally removed in the arg parsing conversion.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rgw: prepare for supporting other meta headers

osdmaptool: fix typo in argument

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

doc: Note that mkcephfs can't do upgrades etc.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: mkcephfs ssh's directly as root@

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

mkcephfs: Config $user is irrelevant when we want to run as root.

It used to just not sudo if $user was empty, on non-ssh do_root_cmd's.
This bug was exposed by commit 47b09e2f0463f6a5ea8cc8e127b012a0cf1ec4af.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Merge branch 'wip-argparse'

Conflicts:
src/rgw/rgw_admin.cc

ceph tool: convert to new-style arg parsing

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ceph_argparse: remove unused macros, functions

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cfuse: use new-style argument parsing stuff

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cmon: use new argument parsing stuff

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cosd.cc: use new argument parsing functions

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

crushtool: use new argument parsing stuff

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw_admin: use new argument parsing stuff

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rbd.cc: use new argument parsing stuff

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add ceph_argparse_withlonglong

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

monmaptool, osdmaptool: fix double-dash handling

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

monmaptool: use new argument parsing stuff

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cmds: remove unused macro

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

TestSignalHandlers: use new argument parsing stuff

Use new argument parsing stuff, remove some deadcode, cleanup.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osdmaptool: use new ceph_argparse stuff

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ceph_argparse: add ceph_argparse_withint

Add an easy way of parsing an int argument. Always match va_start with
va_end.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados.cc: remove unused macro

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados-config: remove unused old-style argparse

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cauthtool: convert to new-style arg parsing

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

gceph tool: convert to new-style arg parsing

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: set perm_mask for swift operations

rgw: fix update_container_stats()

being used when using swift

doc: Explicitly say ceph.conf host= must not include domain name.

Hopefully this will work some day, but it doesn't right now.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

mkcephfs: cosd and cmon setup must happen as root.

Without this, they cannot e.g. write to /var/log, or probably
even to their data directories.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Document mkcephfs-style installation.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

man: mkcephfs: Typo.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

FileStore: close correct fd in version_stamp methods

version_stamp_is_valid and write_version_stamp previously closed op_fd
rather than fd causing write_op_sep to fail in sync_entry.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

FileStore: check write_op_seq return code during sync_entry

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: use the client_messenger for OSD Op replies

We were erroneously using the cluster_messenger instead --
looks like the initial conversion just went wrong since this
is in the repop handling code.

Reported-by: Huang Jin <hjwsm1989@gmail.com>
Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

os/lfnindex: silence compiler warnings

gcc warns about the lack of a space before the ';'!  Crazy.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: only flush imported cap if it is now auth

We can get a cap import that isn't auth if there was a racing import
because the cap since migrated to another node.  We do not want to flush
the caps in that case.  Not to this mds, certainly, and not even to
another, because that would have happened when we got the other import.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Makefile: flock.h!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

cosd: Do filestore conversion after common_init_finish

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: fix include

LFNIndex: Fix lfn parsing for hobjects with keys

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge branch 'hobject_change'

rgw: rename source file

rgw: some var names cleanup

LFNIndex: include index_version in the lfn attribute name

While updating, we use hard links to populate the new directory.  We
need to change the lfn attribute to allow unlinking from the old
directory to continue to work.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: fix parameter passing to swift user creation

PG: Also fix the key when fixing an incomplete log event

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

LFNIndex: Add support generating keyed hobject_t filenames

LFNIndex will now include the key in the filename.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

FileStore,LFNIndex: Bump filestore version

Also adds mechanisms for LFNIndex to handle keyed and keyless hobject_t
versions.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

OSD: fix hobject_t construction

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: Update hobject_t construction

hobject_t now must be supplied with a key during construction.
Also, get_snapset_context must be supplied with the locator key.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

store_test: fix hobject_t construction

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

object.h: Update hobject_t to include object locator key

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: rename openstack to swift

rgw: s/OPENSTACK/SWIFT/g

Merge branch 'wip-flock'

rados tool: fix rados df formatting

rgw: catch buffer::error exceptions thrown at decode

mds: flock: remove auto-insertion of waiting locks.

I don't remember precisely why I had the locking do this (maybe
just for self-containment), but with the messages being re-processed
anyway it's just redundant, and was leading to lots of erroneous
EINTR results.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: user info dumps json/xml

doc: Deb install should work now.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Write about deb installation.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rgw: log of nonexsistent bucket config option

qa: test false

mds: flock: tidy up comments

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

flock: move implementation into a separate c++ file

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

OSD: Fix flipped error codes in do_convertfs

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: poll allocation thread

rgw: user removal delets user index last

and also stop if failed to remove anything, so that
we're not left out with partial user info scattered around.

rgw: multipart meta object uses tmap

doc: Architecture, placeholder in install, and first appendix.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Make object store diagram fit in 1024 pix wide browser.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Move internals into a new section, /dev.

Most of the doc is user-oriented, let's isolate internals more.

Split into multiple files. Clean up RST.

Use Sphinx's graphviz plugin for graphs.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

objectcacher: write vs truncate races are ok

It is possible to write a buffer, truncate it away, and then not have it
when we get the commit.

Fixes: #1467
Signed-off-by: Sage Weil <sage@newdream.net>

mds: truncate sanity checks

Hunting #1472.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: configurable thread pool size

cfuse: initialize Fh to NULL

Avoid debugging confusion.

Signed-off-by: Sage Weil <sage@newdream.net>

client: clean up _{create,release}_fh

Signed-off-by: Sage Weil <sage@newdream.net>

mon: fix 'osd create ...' error paths

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librbd: cosmetic changes to progress stuff

Signed-off-by: Sage Weil <sage@newdream.net>