librgw: implement stat_bucket, call from rgw_lookup

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: annotate unlink's rgw_lookup

An rgw_lookup w/RGW_LOOKUP_FLAG_NONE will return an existing (hence
validated, modulo CREATE races) handle, but fail with ENOENT if none
exists (which is the desired behavior).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add lookup step in rgw_unlink

Implement the required check for underlying leaf object, since with
the current signature it is not known.

(Is that a fixed requirement? If we have previously done a lookup, we
already know the underlying object by name and type.)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: hide uri in RGWListBucketRequest, use full_objectname()

Really would like to remove the requirement to format a URI for these
requests, since it is merely taken apart in parsing before execution.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add marker processing to RGWListBucket(s)Request

Ignores object instances, for now.  Not verified.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: reimplement RGWLibFS::stat_leaf

Now the code attempts to read an object header (which may be cached),
and if this fails, uses RGWStatLeafRequest to find a potential matching
file or directory leaf.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: variant handle subtype and marker cache

RGWFileHandle gets a type-specific variant part.  For the directory
variant, define a flat_map of cookie to marker.  Hash dirent names and
update the marker_cache in readdirs.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: create dir file handles w/correct type

RGW objects which happen to represent directories will always end
in '/', but the trailing slash will not be present in cached versions
of the name.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: implement RGWLibFS::stat_leaf

Stats one of <object>, <object/>.   Additional control over handle
creation has been added to RGWLibFS::lookup_fh.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: incr rework stat

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix uninitialized rgw_fh

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

cohort_lru: fix a maybe uninitialized warning (that seems invalid)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: create explicit directory leaf objects in rgw_mkdir

There is some asymmetry here, since directories are also inferred
from object names containing embedded "/."

The obvious question seems to be, "when can a leaf directory be
garbage collected?"  The answer seems to be, "only explicitly."

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: consolidate RGWLib RGWOps into RGWRequest

The intermediate derivation mirrored that in RGWRest, but we
don't seem to need it.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: don't discard lookup result, fix signedness of min_size

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: make_fhk put base name, object_name in wrong order

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add pseudo() accessor

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add pseudofs awareness to lookup_fh

And then allow rgw_mkdir() to create pseudofs vnodes, when parent
is not the root.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: make_fhk, compute object hashes path-wise

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add depth and make_path to RGWFileHandle

This just (mostly) efficiently builds a full path from its segments
on demand.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: compress RGWFileHandle stat data

Just save the state we actually mutate, and fill stat structures
on demand.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix abuse of std::string& in RGWDeleteObjRequest

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix rgw_unlink

Fix handling of bucket namespace, remove bogus RGWLibFS::is_root()
logic.  Also remove a stray debug check.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: ok, make RGW vnodes cache Unix attrs

Ganesa ULP needs to see a sane progression of attrs in the CREATE
path.

With this change shell actions like

$ echo "hi" > /nfs41/mybucket/mom

work as expected.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: forge Unix attrs in rgw_create

Possibly RGW vnodes should cache Unix attrs, it would reduce the
latency of (frequent) getattr calls, and eventually lines up with
more materialized Unix attributes.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix Unix file modes

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: implement rgw_create + bugfixes

Incremental work to support CREATE,OPEN,WRITE*,CLOSE transactions.
Adds a new tuple-based signature for RGWFileHandle::lookup_fh, and
fixes an inconsisten initialization of fhk and fh.fh_hk.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix size computation in RGWStatObjRequest

Overrides RGWGetObj::execute(), calls the parent method and then
saves obj_size from the manifest.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix many rgw_file return codes

This should all be -(POSIX), or at any rate negative.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: don't abuse fs_private

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: introduce RGW_LOOKUP_FLAG_CREATE

The current API relies on rgw_lookup and rgw_lookup_filehandle to
return file handle instances, and like POSIX, needs to deal in some
way with the NFS/POSIX open-for-create concept.

For now, allow rgw_lookup to return the canonical instance for an
object being created, i.e., if the option RGW_LOOKUP_FLAG_CREATE
is passed.  (I added an RGW_OPEN_FLAG_CREATE also, but it doesn't
do anything.)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add getattr unit tests, restructure

The fids1, fids2 vectors were confusing, as fids2 mapped the
objects in the union of all buckets.

Now there's a matrix of <bucket,object>, and it's trivial to
scan the objects in any one bucket.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: handle a 3 cases in rgw_getattr

Only the object case is mostly correct.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: handle rgw_lookup in the root (minimally)

In addition, check real return code from RGWStatRequest.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: change validity check for rgw_lookup

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add RGWStatObjRequest and use it

* implement a stat operation over RGWGetObj (returning early)
* use stat to validate rgw_lookup
* use stat to implement rgw_getattr

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add diagnostic dump_buckets() call.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fail RGWLibFS::authorize() on user-by-id lookup failure

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: implement rgw_vfsstat

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: rgw_mkdir and rgw_create return addresses

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: pass struct rgw_fh_hk by reference

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: always add struct before rgw_file_handle in rgw_file.h

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: bring back lookup_handle()

Now as a member of RGWLibFS, and exposed via rgw_lookup_handle()
for use by library clients.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw_file_gp: check test state before closing null fh handle

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: implement intrusive filehandle cache

The cache is keyed on a hash of bucket name, plus a hash of the
object name.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add common/cohort_lru.h

This a cache-line-friendly set of primitives for

a) shareded, intrusive maps based on boost::intrusive
b) an LRU system based on them

The system uses per-lane (shard) locks.  Lane count, front cache
size, and lock type are template params.

Initially, use this in preference to an unshareded map or hashtable,
in new RGW code.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw_file: release file handles

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw_file_gp: don't leak a buffer::list

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: checksum verify writev/readv read-after-write

The changes to GET_OBJECT and PUT_OBJECT test options are needed,
since they are (re) writing the same object.

Sadly, READV_AFTER_WRITEV is failing at page offset 7.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: really make hexdump optional, set resid in readv unit test

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix 2 dout problems

* rgw_file.cc needed to include include/assert.h AFTER all rgw
  includes, for some reason

* use *_dout syntax as target of bl.hexdump in lbrgw_file_gp.cc

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: try-implement rgw_readv/rgw_writev

The model in the rgw_readv call is the caller owns the struct rgw_uio,
and uses it to pass uio_offset.

The caller returns the rgw_uio filled out, and provides indirectly
rgw_uio->uio_vio (the full private data is at uio->uio_p1 on return,
the caller (p==private) must not touch it.  The caller returns
private data in rgw_readv_release() when finished.

Meanwhile rgw_writev is atomic.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: improve rgw_write and add WRITE_READ_VERIFY

Update rgw_write to return bytes written as an OUT argument,
use and verify in PUT test.

A new WRITE_READ_VERIFY test writes 16 checksummed, 64K data pages
to the object when --bulk specified.  The returned write value is
checked, but pages not re-read (yet).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: remove junk prints

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: set RGWGetObj::get_data = true, for great justice

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: rework library CephContext* and init

Remove duplicated init step.  Just reference the global
Ceph context for the library instance(s).

The immediate effect of this is to bring the (correctly initialized)
global context into all call paths, which (e.g.) fixes debug
log prints within subsystems.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: dispatch RGWDeleteRequest in object branch in rgw_unlink().

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: implement RGWGetObjRequest

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: declare RGWDeleteObjRequest

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: declare RGWGetObjRequest

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix RGWPutObj s->content_length

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

RGWPutObjRequest: fix get_data()

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw log: disambiguate 'Read AccessControlPolicy' (logs)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add --prelist to gp to prime cache

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix rgw_open param names and invocation

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: avoid illegal rele()

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: don't create/delete in librgw_file.cc (unit test)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fs->root_fh is a pointer (and assign it)

Assign and expose the expanded root_fh indirectly from the mounted
root filesystem structure.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: bucket names can be const

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: reify root handles

Reify root handle, remove traces of legacy (counter) handles.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: incremental RGWPutObj work and almost-complete RGWFH refactor.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

xxHash: build internally

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add ZPage/ZPageSet abstraction (gp unit tests)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

xxHash: add as submodule

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: rgw_file_handle refactor, plus get/put

Implement encapsulation of RGWFileHandle in rgw_file_handle,
so return these from rgw_lookup(), add new rgw_fh_rele() to release
internal state.

Currently the rgw_fh_rele just deletes, in future might be extended
with refcnt support.

Implement the first (atomic) stage of i/o using GET and PUT, add a
new librgw_file_gp test driver for this.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add options for bucket name, multi create, delete

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix acl assignment in RGWCreateBucket_OS_Lib

The logic to create a default S3 policy in
RGWCreateBucket_OS_Lib::get_params() was essentially correct, but
failed to store the resulting policy on the current request.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: don't fake hostname (real hostname isn't used yet)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: don't allow running rgwfile tests as anon

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: partial delete/unlink support

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: return result code from execute_request

Update rgw_mkdir and rgw_readdir to use it.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix uri, read_permissions virtual for create-bucket

* set uri in rgw_mkdir (duh)
* RGWCreateBucketRequest should override RGWHandler::read_permissions
  to return 0

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: block in create bucket, rgw_mkdir

Unit test runs, doesn't...seem to create a bucket.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: re-derive rgw_os_lib family

* shorten names to match files
* remove derivation through rgw_rest intermediates

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: s/rgw_rest_lib.{h,cc}/rgw_os_lib.{h,cc}

There is no longer an explicit (e.g., inherited) dependency on REST
implementations.

The remaining coupling (via RGWLibEnv, and struct req_state), can
be factored out later.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

lirgw: call RGWHanlder_Lib::init_from_header

Call this from the general RGWLibRequest init() method, after
descendant header_init() has run.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: cleanups, incremental ListBuckets fixes

1. remove unused RGWLibRequestEnv code marked to delete
2. restore call to RGWHandler::authorize
3. provide RGWLibRequest::authorize implementing it
   ** the implementation is incomplete, and shows need to expand
      the initial mount auth to deal with keystone, etc
4. since acl check is still not quite right in verify_permissions()
   disable this again (temporarily)
5. fix dout_subsys of rgw_request.cc

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: forward cmdline args to librgw_create

Verified w/-d.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: remove disabled RGWLib RGWREST::get_handler()

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: cleanups in process_request

* remove disabled S3 authorize step (user authorized on mount)
* re-enable op->verify_permission()

Skipping the latter makes RGW unhappy (1s stalls on each request),
but no objects are listing yet.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: pull RGWUserinfo user into RGWLibRequest

Also, return result from process_request(req, io) to caller.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: cache RGWUserInfo in rgw_fs private handle

The private RGWLibFS type holds a RGWUserInfo, and
calls rgw_get_user_info_by_access_key and verifies the returned
secret, checks for suspended user, on rgw_mount.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: split RGWUserInfo from struct req_state

The immediate purpose is to permit auth caching in RGWLib, but
it seems likely that this will be useful for general req_state
de-bloatification.

For now, all request processing paths put a RGWUserInfo on the
stack, so the lifetime of the object is unchanged from when it was
an expanded member.  A later change will introduce caching of
the objects.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: RGWFileHandle and RGWLibFS

Experimentally reframe the C interface objects as subobjects in
private internal types.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fixes for RGWListBucketRequest callback

The cached req_state is still not correct at RGWOp::execute().

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: untested RGWLibListBucketsRequest and caller.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix 2 bugs in RGWListBuckets, remove scaffolding

1. the prior commit had authorize cognate with S3
1.1 which may be right
1.2 but maybe not for each request
1.3 and lacked priming/forged auth data

The forging of auth data is now easy, but the interesting part is
caching and efficiently attaching it so I paused, and

2. disabled auth, which was now overriding and anonymizing all
requests (good!)

3. fixed another instance of reinterpret_cast where (one of)
dynamic_cast or a saved pointer descendant request (avoids a runtime
check, so will add later) is needed

4. fixed bad offset argument to rgw_readdir

The list bucket facility now works, modulo non-auth.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: process_request prologue fixups

* initialize RGWEnv with required HTTP_HOST (incorrectly!)
* assign req_state::cio (not really correctly)
* remove incorrect delete of RGWOp, which no longer alloc'd

Now runs, but don't see buckets.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: reprocess process_request

Refactor RGWLib variant of process_request(...) as a (actually 2,
one which puts an RGWLibIO on the stack) member function of
RGWLibProcess.

Also replace extern abort_early with static inline abort_req(),
since the rgw_rest extern assumes HTTP/REST.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: wire up RGWLibFrontend enqueue and exec operations

Adds the missing bits of RGWLibProcess and RGWLibFrontend, and
calls them from rgw_readdir().

Clients can call enqueue_req() to route a request through the
work queue, or execute_req() to run it inline.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>