rgw_file: fixup attrs across renames

Once Unix attributes were materialized as RGW object/bucket
attributes, it became possible to verify the serialized bucket and
object hashes with the expected values (based on the file-type
invariant path to the object).

Prior to this change, we saved the serialized hash values with the
rest of the Unix attrs, in RGW_ATTR_UNIX1.  Unfortunately, these
values were invalidated by rename operations.  To address this problem,
and to faciliate future operations on objects by their hashed ids,
the id hashes have been moved into their own RGW attr
RGW_ATTR_UNIX_KEY1.  This should allow more efficient search by, but
more importantly to the invalidation problem, allows us to use the
ATTRSMOD_MERGE behavior in RGWCopyObj to get copy offload while still
fixing up the object id.

The code at this commit doesn't use the serialized unix_mode value as
the outgoing st.st_mode result for rgw_getattr operations, pending a
bugfix for Ganesha's expected value range.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: fix silly abuse of flags in fh_lookup

Initially lookup_fh used its cflags argument just to proxy create
flags to new objects.  It's used for call path state now too, so
disambiguate it, and of course, fh->flags can't be consulted for
a call path state.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: don't need to memset Latch

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

cohort_lru: define ctor for TreeX::Latch

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: fix 2 attr decoding issues

* fix general timestamp decoding (changing to a pointer notation)
* capture (std::swap) req_state->bucket_attrs in RGWLibStatBucketRequest

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: force move semantics on rvalue refs in emplace_attrs

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: remove RGWLibFS::mkdir() (no longer called)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: fix nfsns unit test when !created and !create

Avoid dereferencing unitialized top-level objects when !found and the
create option not present (when means we cannot create them).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: check for leaf dir objects and restore their state

This change makes Unix/NFS attributes persistent.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: use emplace_attr in both paths in mkdir2()

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: use emplace_attr in RGWCreateBucket and RGWCopyObj

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: use emplace_attr in RGWPostObj and RGWPutMetadataBucket

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: remove unused label in mkdir2()

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: use emplace_attr in RGWPutObj::execute()

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: encode ux attrs in mkdir2

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: implement new mkdir2 with atomicity

Re-order lookup_fh, FLAG_CREATE and remote create operations,
and provide FLAG_LOCK, to get atomicity.

Other ops which need to linearize at lookup_fh still need to be
converted.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: add encode/decode hooks, emplace_attr

This change implements attr encode/decode glue for RGWGetObj and
RGWPutObj.  Both now have attr maps as member variables.

To save copies, callers producing attrs can use emplace_attr to
move attrs into the map.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: move internals of rgw_read into RGWLibFS::read(...)

Initially, all RGWRequest logic was open coded, but it's all
moving into RGWLibFS to get better encapsulation (e.g., for atomicity).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: declare encoder for Unix object attributes

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file:  declare an attribute for all Unix/rgw_file attrs

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #8559 from liewegas/wip-15643

rgw: fix unsafe c_str() usage

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw/rgw_rados: use to_str() instead of c_str()

bufferlist::c_str() doesn't guarantee null termination.

Fixes: http://tracker.ceph.com/issues/15463
Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add list::to_str()

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8552 from tchaikov/wip-15426

PG: set epoch_created and parent_split_bits for child pg

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8521 from stiopaa1/checkgenerated

check-generated.sh: can't source bash from sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge branch 'jewel'

Merge remote-tracking branch 'origin/greg-fs-testing-411' into jewel

PG: set epoch_created and parent_split_bits for child pg

if we have a wrong "created", OSD receiving an osd_pg_create message
could fail to project its history and crash. a bad split_bits could
mislead the monitor and osd, and hence cause problems.
* set child.info.history.epoch_created to osdmap.get_epoch() instead of
  using the parent's epoch_created
* set child.info.stats.parent_split_bits to the calculated split_bits
  instead using the parent's split_bits

Fixes: http://tracker.ceph.com/issues/15426
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8544 from xiexingguo/xxg-wip-15464

test: fix ut test failure caused by lfn change

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: fix ut test failure caused by lfn change

"https://github.com/ceph/ceph/pull/8496/commits/755c685f2d09c36b53bc39589a77fa73faba5149"
change the lfn_parse_object_name() return type from bool to int but does not
change the relevant test correspondingly.

Fixes: http://tracker.ceph.com/issues/15464
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge branch 'jewel-15402' of git://github.com/ukernel/ceph into greg-fs-testing

Do not ignore standby_for_fscid when selecting a standby.

https://github.com/ceph/ceph/pull/8536

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Fixed ceph-common install.

Fixes: #15438
Signed-off-by: John Wilkins <jowilkin@redhat.com>

librbd: fixed bug in disabling non-primary image mirroring

librbd: fixed bug in disabling non-primary image mirroring

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8496 from athanatos/wip-15312

os/filestore: force lfn attrs to be written atomically, restructure name length limits

mds: take standby_for_fscid into account in FSMap::find_unused()

Fixes: http://tracker.ceph.com/issues/15402
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #8529 from oritwas/wip-rgw-fix-zone-set

radosgw-admin: allow

librbd: Fixed bug in disabling non-primary image mirroring

It is only possible to remove image from rbd_mirroring object if the
mirroring image state is set to DISABLING.

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #8508 from JiYou/fix-ceph-disk

ceph-disk: do not always fail when re-using a partition

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/jewel'

ceph-disk: fix PrepareData.set_type() args should be class member.

Signed-off-by: You Ji <youji@ebay.com>

ceph-disk: fix spelling mistake geattr to getattr.

Signed-off-by: You Ji <youji@ebay.com>

ceph-disk: fix lockbox set_or_create_partition()

Signed-off-by: You Ji <youji@ebay.com>

Merge pull request #8507 from kmroz/wip-rgw-admin-cleanup

rgw-admin: remove unused iterator and fix error message

radosgw-admin: fix name checking

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: allow setting zone when there is no realm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8522 from swamireddy/10.1.1-mailmap

mailmap update

Reviewed-by: Loic Dachary <ldachary@redhat.com>

check-generated.sh: can't source bash from sh

'source' is not sh command and executing bash commands in sh won't
work. We could try to replace it with:
'. ../qa/workunits/ceph-helpers.sh'
but the above file uses bash.

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #8519 from dachary/wip-xenial

tests: add Ubuntu 16.04 xenial dockerfile

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8517 from kjurka/rgw-admin-doc-region-list

doc: rgw admin uses "region list" not "regions list"

Merge pull request #8518 from wjin/fix

crush: fix typo

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #8462 from xiexingguo/xxg-wip-scrub

osd: fix two scrub relevant issues

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8485 from liewegas/wip-fuse-log

ceph-fuse: rotate log file

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #8487 from liewegas/wip-copy-from-fail-cleanup

osd: clean up temp object if copy-from fails

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8486 from liewegas/wip-14695

librados: Revert "rados: Add new field flags for ceph_osd_op.copy_get."

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #7956 from xiexingguo/xxg-wip-fixfstat

common: buffer: put a guard for stat() syscall during read_file

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8516 from runsisi/wip-fix-ret

journal: fix final result for JournalTrimmer::C_RemoveSet

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

tests: add Ubuntu 16.04 xenial dockerfile

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: fix typo

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

doc: rgw admin uses "region list" not "regions list"

journal: fix final result for JournalTrimmer::C_RemoveSet

if the first journal object got removed successfully while later objects
failed then the final result should be set to fail

Signed-off-by: runsisi <runsisi@zte.com.cn>

test/rados/misc.cc: add long locator key and namespace tests

Signed-off-by: Samuel Just <sjust@redhat.com>

LFNIndex: use chain_getxattr_buf

LFN attrs can be longer than these hard coded limits, let's just do
buffer doubling everywhere.

Signed-off-by: Samuel Just <sjust@redhat.com>

chain_xattr: add chain_getxattr_buf

Otherwise callers need to implement buffer doubling in a lot of cases,
which is error prone.

Signed-off-by: Samuel Just <sjust@redhat.com>

LFNIndex::lfn_parse_object_name: return int rather than bool

All current callers (except list_objects) already expect
ret < 0 for error.  Adjust list_objects to skip on EINVAL
instead of 0.

Signed-off-by: Samuel Just <sjust@redhat.com>

LFNIndex::list_objects: lfn_translate does not set errno

Signed-off-by: Samuel Just <sjust@redhat.com>

FileStore::set_xattr_limits_via_conf: add warning if max xattr size smaller than max name

Signed-off-by: Samuel Just <sjust@redhat.com>

test/objectstore/chain_xattr.cc: add test for ensure_single_attr

Signed-off-by: Samuel Just <sjust@redhat.com>

LFNIndex: ensure that lfn and directory attrs are written atomically

We leave skip_chain_cleanup = false since the object may have been
written using code that split the lfn attr over multiple attrs.

Signed-off-by: Samuel Just <sjust@redhat.com>

chain_[f]getxattr: always use size, no reaon to consider CHAIN_XATTR_MAX_BLOCK_LEN

Users of setxattr can now set attrs larger than that size.

Signed-off-by: Samuel Just <sjust@redhat.com>

chain_xattr: s/onechunk/skip_chain_cleanup, add ensure_single_attr

onechunk was a deceptive name since it didn't actually ensure that
a single chunk would be used.  Rename to ensure_single_attr.  Also,
add a parameter to ensure that we use a single attribute.  We need
these to be distinct since we have LFN attrs which have been split
over 254 byte xattrs since after hammer which we need to correctly
clean up.  However, we need to ensure going forward that those attrs
are never split over more than one chunk -- it's not atomic.
Further, skip_chain_cleanup should imply ensure_single_attr, so make
them template params and add a static check to make sure we don't
mess this up in the future.

Signed-off-by: Samuel Just <sjust@redhat.com>

os/,osd/: restructure the rados name length check

Enforce locator length vs the max name length and also
introduce a namespace length limit.

In addition to these checks, also pass the head object to the
ObjectStore implementation to validate.  This allows LFNIndex to account
for the idiosyncracies of its filename escaping and for different xattr
value max sizes.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #8453 from cbodley/wip-rgw-sync-master-down

rgw: retry RGWRemoteMetaLog::read_log_info() while master is down
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

ceph-disk: fix set_data_partition() when data is partition.

Signed-off-by: You Ji <youji@ebay.com>

rgw-admin: fix period delete error message

Signed-off-by: Karol Mroz <kmroz@suse.com>

rgw-admin: remove unused iterator

Signed-off-by: Karol Mroz <kmroz@suse.com>

Merge pull request #8434 from theanalyst/rgw/typo-fix

rgw: fix a typo in error message

Merge pull request #8481 from weiqiaomiao/wq

rgw: the map 'headers' is assigned a wrong value
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8488 from oritwas/wip-15318

rgw: fix problem deleting objects begining with double underscores

rbd-mirror: fixed infinite loop when disabling image mirroringReviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: fixed infinite loop when disabling image mirroring

rbd-mirror: fixed bug that caused infinite loop when disabling image mirroring

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #8501 from dachary/wip-releases

releases: update the v10.1.0 release link

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Luo Kexue name normalization

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Ning Yao affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Vitja Makarov affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

release-notes: v10.1.1 release notes

Signed-off-by: Loic Dachary <loic@dachary.org>

release-notes: v10.1.1 release notes (draft)

cd src/script
virtualenv v
source v/bin/activate
pip install githubpy GitPython requests
GITHUB_ACCESS_TOKEN=xxxxxxx ./ceph-release-notes \
      -r tags/v10.1.0..tags/v10.1.1 $(git rev-parse --show-toplevel)

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: sunspot affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Qinghua Jin affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Richard W.M. Jones affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Ira Cooper affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Star Guo affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Xiaoxi Chen affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Jenkins name normalization

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

releases: firefly was EOL december 2015

Signed-off-by: Loic Dachary <loic@dachary.org>

releases: update the v10.1.0 release link

And add the v10.1.1 release notes as well.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #8500 from linuxbox2/ldap-include

rgw_ldap: make ldap.h inclusion conditional

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge branch 'wip-fuse-log' of git://github.com/liewegas/ceph into greg-fs-testing

Enable log rotation: handle SIGHUP in ceph_fuse.cc and add it to the conf.

https://github.com/ceph/ceph/pull/8485

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge branch 'wip-layout-snafu-followup' of git://github.com/jcsp/ceph into greg-fs-testing

Be more transparent about several failure modes and do more validation on replay

https://github.com/ceph/ceph/pull/8468

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge branch 'jewel-fs-pool-perm' of git://github.com/ukernel/ceph into greg-fs-testing

Extend the existing pool access checking to include specific RADOS namespacse.

https://github.com/ceph/ceph/pull/8444

Reviewed-by: John Spray <john.spray@redhat.com>

Merge branch 'xxg-wip-mds' of git://github.com/xiexingguo/ceph into greg-fs-testing

Fix several message leaks and handle clock skew more politely.

https://github.com/ceph/ceph/pull/8408

Reviewed-by: Greg Farnum <gfarnum@redhat.com>