librgw: fix rpm packaging

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix new Debian control entries

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: remove junk files

Files accidentally added.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: don't compare a string_ref with "" (use empty())

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix boost::basic_string_ref template instantiation

The commit adding Seastar sstring.h replaced a part of the interface
which used an internal string-view type, with an equivalent using
basic_string_ref from Boost.  I relied on a default template argument
not present in older versions.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add debian and rpm packaging

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: consolidate S3 bucket name validation

There is more cleanup potential here, but it is intrusive, as the
validators for bucket and object names have been factored through
the handler inheritance hierarchy.

This change just deletes the S3 validate_bucket_name for now, as
that method was large.  The corresponding object name validator was
not defined in RGWHandler_REST_S3, so I've left it as is, for now.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix copyright in src/include/rados/rgw_file.h

This file originated at Red Hat.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: initialize RGWLib pointer members

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: remove slo, dlo, and supplied-md5 logic in RGWWriteRequest

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: remove RGWLibRequest "magic" numbers added for debugging

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: comment rgw_nfs config_ops.h tunables

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

cohort_lru: remove #if 0 section

The declaration being skipped was meant as a placeholder for future
optimization work, moved to a comment.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

{rgw,common}/Makefile.am: add a few more missing .h files

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

common: add xxhash.h to Makefile.am

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: fix null dereference

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw/rgw: fix librgw.la linkage

Ensure that key symbols from global are satisfied in the library.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw/rgw: reorganize autotools linkage

Adds the RGW-NFS library to the radosgw build, with unit tests.
Adds -fPIC to compile flags to satisfy the linker, but should have
been evident to automake/libtool.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: call obj_rec::sync() on dirs1_b

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw_file_nfsns: create readf_out_name, if needed

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: remove deprecated/unused librgw personality

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: properly detect RGWRados init failure

Unit tests correctly see fail result from librgw_create.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: remove deprecated librgw methods

Remove unused routines:

 * librgw_acl_bin2xml
 * librgw_free_xml
 * librgw_acl_xml2bin
 * librgw_free_bin(librgw_t rgw, char* bin)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: temporary fix RGWWrite

The current behavior of put_data_and_throttle leads to data
curruption, because the supplied buffer is apparently still being
written when the call returns--regardless of the value of
need_to_wait.

For now, do a buffer::copy of the supplied buffer, rather than the
original buffer::static, since that will at least be released
when no longer needed.

Probably this improves apparent rgw_write performance, but is not
what was originally intended.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: cleanup rgw_write

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add new rgw_write unit test

WRITEF1 writes back data wread in READF1.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix RGWReadRequest send_response_data

In hindsight, it seems clear how to consume the available bytes and
account for the bytes written into the caller's buffer.

Verified by unit test on data from S3, with a 6M file read in 1M
chunks, the preferred max read in modern Linux kernel NFS.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add cross-bucket rgw_rename test

It passes.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix intra-bucket rename

Avoid hopeless fail due to binding const::std::string& to a
temporary.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add support for rgw_rename tests

Implements intra-bucket rgw_rename test, which unfortunately fails
currently.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix rgw_unlink and delete atomicity

Move internal unlink logic into new RGWLibFS::unlink method, fix
logic for files and directories.

Use RGWFileHandle::mtx and  RGWFileHandle::FLAG_DELETED to ensure
atomicity of handle creates and deletes.

Remove handles for unlinked objects from cache.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: implement object rename

The implementation uses currently does a copy, followed by a
delete operation (if successful).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix timestamp on create (another case)

In the rgw_create regular file case, the timestamp and size update
logic (though present) was being bypassed because the "FLAG_CREATE"
indicator which formerly propagated up from RGWLibFS::lookup_fh
was lost in the LRU change, which refactored the code for creating
new RGWFileHandle objects.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: cleanup, move 3 prints to log channel

Save a pointer to CephContext in RGWProcess for use by logger--
using the existing pointer in ThreadPool would be an intrusive
change.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add missing RGWFileHandle::stat in rgw_mkdir

The lack of stat here induced a serious memory corruption, due to
the organization of Ganesha file attributes (and their reliance on
the FSAL to initialize the memory).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix leading-slash special case, again

Fix boilerplate to sanitize relative_object_name() in rgw_create
as well as the RGWStatObjRequest case of RGWLibFS::stat_leaf.
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: disable name-cache lookups w/CREATE

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix a prefix bug and bucket timestamp bug

The fix for missing bucket.creation_time in RGWStatBucket is not
complete--we're not carrying all the info over.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: ignore "/" as a common_prefix

This happens commonly, apparently when a parent object name ends
with '/' (as all explicit directories do).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: rgw_getattr refactor, RGWLibFS::stat_leaf (object)

Rewrite rgw_getattr to just dispatch into RGWLibFS::getattr, where
we can better use RGWFileHandle and RGWLibFS internals.

Initially, this change intended to dispatch a new RGWStatObjRequest
in the file case, but during implementation, discovered that in
(apparently) all file cases, it has already been done--but the path
had stopped working with the naming change.

Current GETATTR_DIRS1 unit test, when repeated so relying on lookup
(rather than saving size after rgw_write/rgw_close), passes.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix rgw_read, add unit test

rgw_read had been trivially broken by the relative path change,
fixed.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add rgw_getattr tests

Re-implement rgw_getattr to just dispatch into RGWFileHandle::stat,
we expect all objects to stat correctly for their type.

The size of file objects is currently reporting as 0, where we expect
a small, positive size (reviewing).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: re-implement rgw_create

rgw_create hadn't been re-implemented for NFS namespace behavior,
where we expect it to create empty file objects.

It needs to do a full name conflict check with prefix-matching,
to catch conflicts within object names.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: update rgw_* api calls so that all take a flags arg

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: normalize obj_rec::sync calls, fix signedness of refcnt asserts

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: partial getattrs/gc checks

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix handle type in rgw_mkdir, finish dirs1 create tests

Fixes handle type after rgw_mkdir.

Add handle type checks to nfsns unit tests.  Adds tests to create
files and directories with non-bucket parent.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: set directory ctime and mtime

Adds method to RGWStatBucketRequest to access the creation_time of
the returned bucket.

(For now, that's the only timestamp we have.)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: refactor nfsns dirs1 and fix a "bad unlink" case

Fix the error returns from rgw_unlink.  This is sufficient to
catch the errors that arise from operation failure.

We won't notice attempts to remove interior directories--and for
various reasons, that may be problematic (skip for now).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: incremental bucket-object 2-level tests

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: enforce S3 object name restrictions

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: enforce S3 bucket name restrictions

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: FLAG_CREATE cleanup

Split the transiently-creating concept (CREATING) from the request
to create objects used in lookup_fh.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: nfsns dirs1 delete, make hier1 conditional

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: incr. fix various nfs ops (REBASEME)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: reorg SETUP_ENUMERATE test

Stat nfsroot bucket, create it only if !exist && do_create.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: conditionally send '.' and '..' (rgw_readdir)

Ganesha traditionally does not expect them, but they may be needed
when bypassing the MDCACHE (2.4).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: stop abusing parents

Also make sure that root file handle has directory as its variant
type.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: likely correct marker handling

New logic propagates is_truncated traveral-finished indicator up
from bucket and object list operations in all cases, and correctly
expands relative paths.

Unit test output looks promising.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: is_truncated experiments

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw nfsns:  create/delete updates

Add ability to delete marker objs, and create (but not yet delete)
nfs_hier objects.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: don't early terminate readdirs (in progress)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: remove unused st

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: split nfsns bucket and object create blocks

Helps to see the respective timings.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw_file_nfsns: bugfix MARKER1_SETUP

Successfully creates 5*1024 objects at /nfsroot/nfs_marker/, verifies
externally.

Follow-up rgw_readdir shows 999 results, apparently the initial readdir
segment works correctly, but restart from offset fails.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: consolidate dup'd librgw, rgwlib

There can be only one.  Fixes crash on shutdown.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: remove search_prefix()

Since it turns out this is a synonym for relative_object_name().

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

incremental marker

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: omit always-false depth test.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: rework full_object_name() and friends

Flex RGWFileHandle::full_object_name() to build either a search
prefix (omits leading "/" and bucket name) or else the full path
to the root (e.g., for pretty-printing), depending on the value
of a boolean omit_bucket argument.

New convenience methods search_prefix() and relative_object_name()
select these, respectively.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add continued readdir test

Verify that files created in a single, large directory are correctly
enumerated through a sequence of rgw_readdir() calls, at successive
offsets checking cookie/marker and bounds invariants.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

marker

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Fix copyright statements on new test driver files (Red Hat, Inc.)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: dir traversal state GC added

To minimize overhead, traversal events are modeled as a linear
sequence of fh_key, timestamp tuples.  GC operates out-of-line in
LIFO order, and includes a ref cycle.  If the matching directory
object can't be found by handle, it's already gone, and nothing
happens.

To avoid lock contention, scanning of the event sequence is done
in groups of 24 by default, but in groups of sequence/4 if the
event sequence grows larger than 500 events.

Push READDIR events on successful component readdir (bucket or
object listing partial result, as appropriate).

Revise locking, timestamp and update link count in all cases.  Link
counts are approximate (metadata to represent it is not stored).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: restore readdir prints (move to dout)

They are useful for checking the exact strings being produced,
and in which sequence (object or common prefix).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: move readdir operations into RGWFileHandle::readdir

The motivation here is to localize state updates to the class,
e.g., to hide consistency logic.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: implement small-directory name cache

Augment the existing marker cache with cache of name->type
mappings for up to 128 dirents (appx. 4x the common upper-bound
of 32).

The motivation here is to use the name and type information from
readdir, rather than discarding it.  Boosts perf. at least 60%
on unit test corpus.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: remote redundnat prints in RGWReaddirRequest

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: don't increment offset at rcb (it is no longer a counter)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fill in dirent cache

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: implement framework for fs periodic work

This can be used for, e.g., gc of stale handles and directory
traversals.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add intrusive refcnt to RGWLibFS

Ensures that file system instances cannot be disposed while (e.g.)
background processing is ongoing.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: header and namespace reorganization

Move RGWLibProcess and RGWLibFrontend into new rgw_lib_frontend.h
header, which can include rgw_lib.h and rgw_file.h.

This allows the already-defined RGWLibProcess to call methods on
RGWLibFS objects (e.g., to perform GC, which follows in a separate
commit).

In addition, move all of the library and file interface into
namespace rgw.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: use string_ref remove_prefix

Save a few cycles over re-assigning a string_ref suffix substring
to itself.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix initialization of string_ref from temporary

Found by valgrind.  The good news:  amounts to proof that the
sref notation works.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

cohort_lru: use correct new/delete operator pair

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: move ObjUnref trace print ahead of...unref

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: cleanup add_marker and handlers

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: rename RGWGetObjRequest, RGWListBucketRequest

The behavior of these requests is basically specific to the
needs of NFS READDIR and READ.  (RGWWriteRequest already updated.)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

common: add (tweaked) sstring.h

Add sstring.h from Seastar, slightly adapted for c++2011 and
boost::string_ref.

An sstring resembles std::string, but the size of its small-string
optimization (static allocation) is controlled by a template
argument (cf. small_vector).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix RGWLibFS::stat_leaf ex-ref

Found verifying refcounts at RGWLibFS::close.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

lirgw: call RGWLibFS::close in rgw_umount

Also remove an unused warning in currently-disabled rgw_readv.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: implement LRU lifecycle for handles

This re-uses intrusive lane LRU and co-resident intrusive b-tree
for lookups (formerly, we were using just the dictionary).

Machinery for cache drain on unmount/release of an active filesystem
instance has been added.  Currently, this code won't be reached.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: remove dead code (rgw_readdir)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: use correct finish path for RGWWriteRequest (logging and stats)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: update posix-style read path

Update RGWGetObj to take ulp buffer rather than transferring the
elements of a buffer::list and re-traversing it.

This breaks the readv variant, but that was not final.  What it
fixes is offset handling.

Also, skip CREATE_BUCKET if not requested, in librgw_file_aw.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: set iov->iov_len to nread (get)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: sequential rgw_write works

This commit finishes hooking up RGWWriteRequest, and its supporting
RGWLibContinuedRequest.

The rgw_read call now fails--trivially, because it never handled
starting offset correctly.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: update mtime on write finish

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: -EISDIR

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>