pybind: rados cleanup

This change uses:
* a pystring to store ioctx state like other state attributes and like
  previous binding
* use __dealloc__ instead of __del__ to ensure Object are freed correctly

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Merge pull request #7684 from ceph/wip-pybind-install

pybind: use correct subdir for rados install-exec rule

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

pybind: use correct subdir for rados install-exec rule

This fixes package builds and 'make install'

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7621 from sileht/sileht/rados-cython

python binding of librados with cython

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Update the documentation

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

librbdpy: Use new rados lib

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

ceph.in: Use new python rados module

This implements run_in_thread inside the ceph command itself.

And fixes the ceph command bootstrap when it run inside the
source tree to correctly load the new rados python module.

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Fix rpm/deb packaging

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Remove old rados pybinding

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Cython Rados module

Notable changes:

* run_in_thread have disapeared
* timeout argument of some methods are ignored
* rados_create_write_op/rados_create_read_op returns WriteOp/ReadOp
  instead of the pointer address
* rados_monitor_log callback 'arg' arguments was broken in previous python
  binding (callback was called with the pointer address instead pointed object)
* object attributes that was pointer addresses are now private and not accessible in python

Some tests have been added to cover all methods

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Merge pull request #7603 from roidayan/xio_fixes

Xio fixes

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Move pybind rbd module into it own directory

To allow to create a autonomous rados module with cython.
We move the current librbdpy to the rbd sub directory.

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Merge pull request #7520 from renhwsky/renhw-wip-mon-monitor

mon/monitor: some clean up

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7629 from ceph/wip-14750-op-timeouts

Objecter: detect laggy ops with objecter_timeout, not osd_timeout

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #7655 from gregsfortytwo/wip-metarequest-lock

client: removed unused Mutex from MetaRequest

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #7616 from gregsfortytwo/wip-timer-14697

mds: don't double-shutdown the timer when suiciding

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #7490 from x11562/mds_mdstype_ref

mds: function parameter 'df' should be passed by reference

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7610 from dillaman/wip-bit-vector

common/bit_vector: use hard-coded value for block size

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #7653 from dillaman/wip-librbd-race-conditions

librbd: several race conditions discovered under single CPU environment

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7408 from ceph/wip-14549

rgw: don't use s->bucket for metadata api path entry

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #7633 from zaitcev/cleanup-various-1

rgw: cleanups to comments and messages

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

librados_test_stub: watch_flush should block until notifies complete

It shouldn't block until all the acks are received as that is not
how librados handles the method.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: lock notifications should be executed outside librados thread

Otherwise it's possible that the notification will be blocked if
the librados AIO thread isn't available to invoke the notification
handler.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: potential race on image close

If multiple states are pending, it's possible for an image to
be closed while the state lock is unlocked during the callback.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: potential deadlock during AIO image re-open

The AIO image open might complete under the librados AIO callback.
Attempting to close the old image handle under that thread will
deadlock under the new librados_test_stub flush handling.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: ensure AIO callback is also flushed

There are sporadic librbd unit test failures due to the
mock being destroyed while an active librados_test_stub
AIO callback is completing.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge branch 'wip-commit_position-tests' of git://github.com/trociny/ceph

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

tests: librbd: commit position updated after journal replay

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

common/bit_vector: use hard-coded value for block size

The CEPH_PAGE_SIZE is not actually a constant. On aarch64 platforms
the page size is 64K instead of the assumed 4K.  The CRC block
computations require a true constant.

Fixes: #14747
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7646 from dachary/wip-ceph-disk-flake8

ceph-disk: flake8 fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-disk: flake8 fixes

Signed-off-by: Loic Dachary <loic@dachary.org>

rgw: cleanups to comments and messages

These are minor cleanups that do not affect the code in a
material way:
- Forgot to add --tenant to usage message
- Make the help slightly less misleading
- Fix a strange error message "could not add remove caps"
- Spelling in a comment
- The vstart.sh can be run with civetweb

Signed-off-by: Pete Zaitcev <zaitcev@redhat.com>

client: removed unused Mutex from MetaRequest

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7634 from ceph/rgw-nfs-5

nfs for rgw (Matt Benjamin, Orit Wasserman)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Objecter: detect laggy ops with objecter_timeout, not osd_timeout

Fixes: 14750
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7350 from ddiss/tracker14499_enotconn_rados_monitor_log

librados: check connection state in rados_monitor_log

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7584 from ifed01/wip-14511

osd: fix lack of object unblock when flush fails

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7593 from ifed01/wip-no-blockedpromote-return-fix

osd: fix return value from maybe_handle_cache_detail()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7587 from cbodley/wip-pageset-align

osd: memstore: fix alignment of Page for test_pageset

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #7595 from liewegas/wip-hobject-stringify

osd: ghobject_t: use ! instead of @ as a separator

Merge pull request #7596 from dachary/wip-erasure-code-corpus

tests: sync ceph-erasure-code-corpus for mktemp -d

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7608 from liewegas/wip-warnings

tests: fix a few build warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>

librgw: fix rpm packaging

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix new Debian control entries

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: remove junk files

Files accidentally added.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: don't compare a string_ref with "" (use empty())

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix boost::basic_string_ref template instantiation

The commit adding Seastar sstring.h replaced a part of the interface
which used an internal string-view type, with an equivalent using
basic_string_ref from Boost.  I relied on a default template argument
not present in older versions.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add debian and rpm packaging

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: consolidate S3 bucket name validation

There is more cleanup potential here, but it is intrusive, as the
validators for bucket and object names have been factored through
the handler inheritance hierarchy.

This change just deletes the S3 validate_bucket_name for now, as
that method was large.  The corresponding object name validator was
not defined in RGWHandler_REST_S3, so I've left it as is, for now.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #7626 from tchaikov/wip-fix-automake

Makefile: workaround an automake bug for "make check"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

librgw: fix copyright in src/include/rados/rgw_file.h

This file originated at Red Hat.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: initialize RGWLib pointer members

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: remove slo, dlo, and supplied-md5 logic in RGWWriteRequest

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: remove RGWLibRequest "magic" numbers added for debugging

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: comment rgw_nfs config_ops.h tunables

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

cohort_lru: remove #if 0 section

The declaration being skipped was meant as a placeholder for future
optimization work, moved to a comment.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

{rgw,common}/Makefile.am: add a few more missing .h files

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

common: add xxhash.h to Makefile.am

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: fix null dereference

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw/rgw: fix librgw.la linkage

Ensure that key symbols from global are satisfied in the library.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw/rgw: reorganize autotools linkage

Adds the RGW-NFS library to the radosgw build, with unit tests.
Adds -fPIC to compile flags to satisfy the linker, but should have
been evident to automake/libtool.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: call obj_rec::sync() on dirs1_b

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw_file_nfsns: create readf_out_name, if needed

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: remove deprecated/unused librgw personality

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: properly detect RGWRados init failure

Unit tests correctly see fail result from librgw_create.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: remove deprecated librgw methods

Remove unused routines:

 * librgw_acl_bin2xml
 * librgw_free_xml
 * librgw_acl_xml2bin
 * librgw_free_bin(librgw_t rgw, char* bin)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: temporary fix RGWWrite

The current behavior of put_data_and_throttle leads to data
curruption, because the supplied buffer is apparently still being
written when the call returns--regardless of the value of
need_to_wait.

For now, do a buffer::copy of the supplied buffer, rather than the
original buffer::static, since that will at least be released
when no longer needed.

Probably this improves apparent rgw_write performance, but is not
what was originally intended.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: cleanup rgw_write

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add new rgw_write unit test

WRITEF1 writes back data wread in READF1.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix RGWReadRequest send_response_data

In hindsight, it seems clear how to consume the available bytes and
account for the bytes written into the caller's buffer.

Verified by unit test on data from S3, with a 6M file read in 1M
chunks, the preferred max read in modern Linux kernel NFS.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add cross-bucket rgw_rename test

It passes.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix intra-bucket rename

Avoid hopeless fail due to binding const::std::string& to a
temporary.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add support for rgw_rename tests

Implements intra-bucket rgw_rename test, which unfortunately fails
currently.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix rgw_unlink and delete atomicity

Move internal unlink logic into new RGWLibFS::unlink method, fix
logic for files and directories.

Use RGWFileHandle::mtx and  RGWFileHandle::FLAG_DELETED to ensure
atomicity of handle creates and deletes.

Remove handles for unlinked objects from cache.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: implement object rename

The implementation uses currently does a copy, followed by a
delete operation (if successful).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix timestamp on create (another case)

In the rgw_create regular file case, the timestamp and size update
logic (though present) was being bypassed because the "FLAG_CREATE"
indicator which formerly propagated up from RGWLibFS::lookup_fh
was lost in the LRU change, which refactored the code for creating
new RGWFileHandle objects.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: cleanup, move 3 prints to log channel

Save a pointer to CephContext in RGWProcess for use by logger--
using the existing pointer in ThreadPool would be an intrusive
change.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add missing RGWFileHandle::stat in rgw_mkdir

The lack of stat here induced a serious memory corruption, due to
the organization of Ganesha file attributes (and their reliance on
the FSAL to initialize the memory).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix leading-slash special case, again

Fix boilerplate to sanitize relative_object_name() in rgw_create
as well as the RGWStatObjRequest case of RGWLibFS::stat_leaf.
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: disable name-cache lookups w/CREATE

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix a prefix bug and bucket timestamp bug

The fix for missing bucket.creation_time in RGWStatBucket is not
complete--we're not carrying all the info over.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: ignore "/" as a common_prefix

This happens commonly, apparently when a parent object name ends
with '/' (as all explicit directories do).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: rgw_getattr refactor, RGWLibFS::stat_leaf (object)

Rewrite rgw_getattr to just dispatch into RGWLibFS::getattr, where
we can better use RGWFileHandle and RGWLibFS internals.

Initially, this change intended to dispatch a new RGWStatObjRequest
in the file case, but during implementation, discovered that in
(apparently) all file cases, it has already been done--but the path
had stopped working with the naming change.

Current GETATTR_DIRS1 unit test, when repeated so relying on lookup
(rather than saving size after rgw_write/rgw_close), passes.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix rgw_read, add unit test

rgw_read had been trivially broken by the relative path change,
fixed.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: add rgw_getattr tests

Re-implement rgw_getattr to just dispatch into RGWFileHandle::stat,
we expect all objects to stat correctly for their type.

The size of file objects is currently reporting as 0, where we expect
a small, positive size (reviewing).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: re-implement rgw_create

rgw_create hadn't been re-implemented for NFS namespace behavior,
where we expect it to create empty file objects.

It needs to do a full name conflict check with prefix-matching,
to catch conflicts within object names.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: update rgw_* api calls so that all take a flags arg

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: normalize obj_rec::sync calls, fix signedness of refcnt asserts

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: partial getattrs/gc checks

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: fix handle type in rgw_mkdir, finish dirs1 create tests

Fixes handle type after rgw_mkdir.

Add handle type checks to nfsns unit tests.  Adds tests to create
files and directories with non-bucket parent.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: set directory ctime and mtime

Adds method to RGWStatBucketRequest to access the creation_time of
the returned bucket.

(For now, that's the only timestamp we have.)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: refactor nfsns dirs1 and fix a "bad unlink" case

Fix the error returns from rgw_unlink.  This is sufficient to
catch the errors that arise from operation failure.

We won't notice attempts to remove interior directories--and for
various reasons, that may be problematic (skip for now).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: incremental bucket-object 2-level tests

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: enforce S3 object name restrictions

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: enforce S3 bucket name restrictions

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: FLAG_CREATE cleanup

Split the transiently-creating concept (CREATING) from the request
to create objects used in lookup_fh.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: nfsns dirs1 delete, make hier1 conditional

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: incr. fix various nfs ops (REBASEME)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: reorg SETUP_ENUMERATE test

Stat nfsroot bucket, create it only if !exist && do_create.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librgw: conditionally send '.' and '..' (rgw_readdir)

Ganesha traditionally does not expect them, but they may be needed
when bypassing the MDCACHE (2.4).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>