Client: check dir is still complete after dropping locks in _readdir_cache_cb

We drop the lock when invoking the callback, which means the directory
we're looking at might get dentries trimmed out of memory. Make sure that
hasn't happened after we get the lock back. If it *has* happened, fall back
to requesting the directory contents from the MDS. Update the dirp location
pointers after each entry to facilitate this.
Because this requires we update the dirp->at_cache_name value on every loop,
we rework the updating scheme a bit: to dereference the dn->name before
unlocking, so we know it's filled in; and since we update it on every loop
we don't need to refer to the previous dentry explicitly like we did before.

This should also handle racing file deletes: we get back a trace on
the removed dentry and that will clear the COMPLETE|ORDERED flags.

Fixes #12297

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5245 from yuyuyu101/wip-async-fix-18

AsyncConnection: Fix incorrect sign_message usages

Merge pull request #5244 from zhouyuan/rgw_obj_stripe_size

rgw: doc: Fix radosgw stripe size config ref

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #5240 from dachary/wip-ceph-release-notes

tools: src/script/ceph-release-notes normalization

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #5165 from zhouyuan/librados_default_crush_ruleset

Make librados pool_create respect default_crush_ruleset

Reviewed-by: Kefu Chai <kchai@redhat.com>

librados: Make librados pool_create respect default_crush_ruleset

Make pool_create in librados use the osd_pool_default_crush_replicated_ruleset

Fixes: #11640
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

AsyncConnection: Make sign_message ahead of construct message bufferlist

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

AsyncConnection: Fix non-fastdispatch message doesn't prepare case

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

rgw: doc: Fix radosgw stripe size config ref

s/rgw object stripe size/rgw obj stripe size
https://github.com/ceph/ceph/blob/master/src/common/config_opts.h#L1024

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: src/script/ceph-release-notes normalization

Issue an error if for the title does not start with a known prefix.
Issue an error if no issue is associated to a pull request.

Group the pull requests fixing the same issue together.

Add link to the issue and the pull request.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #5230 from majianpeng/check_ops_in_flight_cleanup

common: clean up code for OpTracker::check_ops_in_flight.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

common: clean up code for OpTracker::check_ops_in_flight.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #5220 from vumrao/wip-vumrao-12318

doc: change tcp rcvbuf and tcp nodelay to ms tcp rcvbuf and ms tcp nodelay

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: change tcp rcvbuf and tcp nodelay to ms tcp rcvbuf
     and ms tcp nodelay

Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>

Merge pull request #4939 from ceph/wip-offline-backward

cephfs-data-scan (offline backward file recovery)

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5190 from guce/master

radosgw-admin: after subuser modify print only once user info

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5183 from zhouyuan/rgw_max_put_size_configurable

rgw: Make RGW_MAX_PUT_SIZE configurable

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5213 from wido/wip-rgw-keepalive

rgw: If the client sends a Connection: close header respond accordingly.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5210 from tweenk/docs-improvements

Document librbd::parent_spec and librbd::parent_info.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5215 from SUSE/wip-fdupes

ceph.spec.in: do not run fdupes, even on SLE/openSUSE

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ceph.spec.in: do not run fdupes, even on SLE/openSUSE

In openSUSE there is a policy to use %fdupes in the spec file if RPMLINT
complains about duplicate files wasting space in the filesystem.

However, RPMLINT is not so complaining, so drop fdupes.

http://tracker.ceph.com/issues/12301 Fixes: #12301

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5102 from ceph/wip-mds-refactor-small

Minor MDS:: refactor

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5209 from zhouyuan/rgw_swift_user

rgw: Make vstart.sh print out swift user info

Reviewed-by: Sage Weil <sage@redhat.com>

packaging: add cephfs-data-scan

Signed-off-by: John Spray <john.spray@redhat.com>

tools/cephfs: add cephfs-data-scan

For scraping cephfs data pools for metadata in
backtraces, and injecting that metadata into
the cephfs metadata pool (initially offline).

Signed-off-by: John Spray <john.spray@redhat.com>

doc: add some docs about cephfs-data-scan

These are deliberately fairly sparse, because:
 * These tools are for experts
 * These tools may well be wrapped in a higher
   level recovery tool that orchestrates parallel
   workers at some stage.

Signed-off-by: John Spray <john.spray@redhat.com>

include/ceph_fs.h: define magic LOST+FOUND ino

So that recovery tools can create this dir without
having to consult/modify InoTable.

Signed-off-by: John Spray <john.spray@redhat.com>

cls: add CephFS object class

For use in cephfs-data-scan for file size and mtime
recovery, by doing a set-if-greater for size/mtime
xattrs on the 0th object of a file.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: include damage_flags in inter-mds encoding

...and correspondingly bump the cluster protocol
version as these structures are not versioned
themselves.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: bump required version on fnode_t encoding

So that older MDS instances can't miss/ignore
the damage flags.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add damage_flags_t to inode+frag

Used to mark inodes+fnodes with flags indicating
that particular types of damage have either
been detected (scrub/fetch) or caused (cephfs-data-scan
inject).

This should serve as a cue to the MDS to do online
repair of stats for the nodes that have been injected.

Signed-off-by: John Spray <john.spray@redhat.com>

mds/CInode: move hash_dentry_name up into InodeStore

Because it only depends on the data in the store, not
anything from a live MDS, and it's useful when
we're dealing with fragmented directories from
an offline tool.

Signed-off-by: John Spray <john.spray@redhat.com>

tools: fix journal reset error handling

When we failed and prompted for --force, it still
looked like a success.  ::reset never had any
proper error handling since it was lifted out
of the MDS, so add it now.

Signed-off-by: John Spray <john.spray@redhat.com>

client: fix typo

s/bye/byte/

Signed-off-by: John Spray <john.spray@redhat.com>

mds: correct typo in log message

s/issueing/issuing/

Silly, but it annoyed me every time I saw it...

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #5164 from ceph/wip-12088

client: reference counting 'struct Fh'

Reviewed-by: John Spray <john.spray@redhat.com>

rgw: If the client sends a Connection: close header respond accordingly.

HTTP/1.1 assumes Keep-Alive by default, but if a Connection: close header is send
the server should respond with it as well.

This makes the client close the connection after the request.

Fixes: #12298

Merge pull request #5201 from ceph/wip-12272

tools: Fix rados export to use io_ctx in the right way

Reviewed-by: John Spray <john.spray@redhat.com>

rgw: Make vstart.sh print out swift user info

Currently vstart.sh only print out S3 user info. This patch updates it
to print swift user info too. Also changes the swift account/user/password
to test/tester/testing which align with openstack-swift better:
https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L270

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Fixes: #12286 radosgw-admin: after subuser modify print only once user info.
remove rgw_admin.cc OPT_SUBUSER_MODIFY, show_user_info code block.

switch (opt_cmd) {
...
case OPT_SUBUSER_MODIFY:
show_user_info(info, formatter);       //show first time (remove this)
break;
...
}

// output the result of a user operation
if (output_user_info) {
...
show_user_info(info, formatter);       //show second time
}

test fix:
before: after subuser modify print twice user info.
after changes, do the same procedure, print only once user info.

Signed-off-by: guce guce@h3c.com

rgw: Make RGW_MAX_PUT_SIZE configurable

The 5GB limit of a single operation uploading was part of S3 spec.
However some private setups may have some special requirements
on this limit. It's more convinent to have a configurable value.

Fixes: #6999
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: Fix rados export to use io_ctx in the right way

Cause by: 868f6b7626db4a802b38d1d8ce5135108b322f74

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #5197 from ceph/wip-cot-meta

Wip cot meta

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5198 from tchaikov/wip-doc-erasure-fix

doc/erasure-code: fix couple typos

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/erasure-code: fix couple typos

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools, test: Some ceph-objectstore-tool error handling fixes

Improve various error messages generated with invalid syntax
Add test cases for most of these error messages

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add debug argument to the ceph-objectstore-tool test

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Check for valid --op earlier so we can get a better error message

Signed-off-by: David Zafman <dzafman@redhat.com>

tools, test: Add ceph-objectstore-tool to operate on the meta collection

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Fix newlines in output of --op list

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Fix dump-super which doesn't require pgid

A valid pgid had to be specified in order for superblock to be displayed,
but it was otherwise ignored.
Fix message that didn't include dump-super

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Check and specify commands that require the pgid specification

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #4404 from rzarzynski/wip-11431

rgw: fix lack of account name attribute in XML-formatted bucket listing of Swift account

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5078 from rzarzynski/wip-12158

rgw: send Content-Length in response for GET on Swift account.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge branch 'wip-temp'

Conflicts:
src/osd/ReplicatedPG.cc
src/test/osd/RadosModel.h

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

osd, tools: Always filter temp objects since not being exported

Can't use object_locator_to_pg() directly with a temp object
because the pool is negative.

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Don't export temporary objects until we have persistent-temp objects

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: fix temp clearing in OSD

Signed-off-by: David Zafman <dzafman@redhat.com>

Document librbd::parent_spec and librbd::parent_info.

Add an explanatory comment to parent_types.h that explains what
the "overlap" member in parent_info means. This was not exactly easy
to figure out.

Signed-off-by: Krzysztof Kosiński <krzysztof.kosinski@intel.com>

Merge pull request #5175 from david-z/wip-mds-dump-crash

Fix mds dump_ops_in_flight crashing ocassionally

Fix mds dump_ops_in_flight crashing ocassionally

Merge pull request #4977 from theanalyst/rados/wip-9737

rados/tool: handle --snapid correctly

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5185 from yuyuyu101/wip-store-test-zero

StoreTest: Add zero test for SyntheticTest

Merge pull request #3273 from majianpeng/fix4

cleanup code for RecoveyMessages

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #4143 from majianpeng/ec-notsupp-omap

fix bugs about cache tier.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4329 from majianpeng/filestore-copy

Filestore copy:using splice instead read/write for clone

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #4458 from wonzhq/dup-promote

osd/ReplicatedPG: need to queue op when avoiding dup promotion in maybe_handle_cache

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #5075 from SUSE/wip-logrotate-requires

packaging: add find and which dependencies

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

packaging: add find and which dependencies

The postrotate script in src/logrotate.conf uses the which and find utilities
to do its work. Although logrotate itself is only a Recommends, I think which
and find are so ubiquitous that it makes sense to have them as hard
dependencies.

Also, I checked and the package names which and findutils are the same on all
the RPM distros we are currently targeting in the spec file.

In Debian, find is also in a package called findutils while the package
containing 'which' is called debianutils.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5187 from SUSE/wip-12261

ceph.spec.in: install 95-ceph-osd.rules, mount.ceph, and mount.fuse.c…

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #5181 from SUSE/wip-snappy-devel

ceph.spec.in: snappy-devel for all supported distros

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #5180 from SUSE/wip-add-missing-specfile-bit

ceph.spec.in: add missing -%{release}

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #5076 from SUSE/wip-logrotate-hard-dependency

packaging: make logrotate a hard dependency across all distros

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
 Conflicts:
debian/control

ceph.spec.in: install 95-ceph-osd.rules, mount.ceph, and mount.fuse.ceph properly on SUSE

http://tracker.ceph.com/issues/12261 Fixes: #12261

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5079 from SUSE/wip-12166

ceph.spec.in: use %_udevrulesdir to eliminate conditionals

Merge pull request #5126 from SUSE/wip-12201

ceph.spec.in: /var/run/ceph fixes

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

ceph.spec.in: use _udevrulesdir to eliminate conditionals

The conditionals governing where 50-rbd.rules is installed were not doing the
right thing on SUSE distros.

Start using the %_udevrulesdir RPM macro, while taking care that it is defined
and set to the right value. Use it to eliminate some conditionals around other
udev rules files as well.

http://tracker.ceph.com/issues/12166 Fixes: #12166

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5172 from ceph/wip-rm-git-build-dep

Remove git build-time dependency

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5167 from zaitcev/wip-rgw-doc-layout

rgw: Document the layout of pools and objects

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5184 from majianpeng/compiler-fix

Compiler fix warning message

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rgw: fix signed/unsigned compare warning.

rgw/rgw_op.cc: In member function 'virtual void
RGWCompleteMultipart::execute()':
rgw/rgw_op.cc:3133:27: warning: comparison between signed and unsigned
integer expressions [-Wsign-compare]
   if (parts->parts.size() > s->cct->_conf->rgw_multipart_part_upload_limit) {

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

test: fix signed/unsigned compare warning.

 In file included from test/objectstore/chain_xattr.cc:29:0:
../src/gmock/gtest/include/gtest/gtest.h: In instantiation of
‘testing::AssertionResult testing::internal::CmpHelperEQ(const char*,
const char*, const T1&, const T2&) [with T1 = long unsigned int; T2 =
int]’:
../src/gmock/gtest/include/gtest/gtest.h:1484:23:   required from
‘static testing::AssertionResult
testing::internal::EqHelper<lhs_is_null_literal>::Compare(const char*,
const char*, const T1&, const T2&) [with T1 = long unsigned int; T2 =
int; bool lhs_is_null_literal = false]’
test/objectstore/chain_xattr.cc:205:314:   required from here
../src/gmock/gtest/include/gtest/gtest.h:1448:16: warning: comparison
between signed and unsigned integer expressions [-Wsign-compare]
   if (expected == actual) {

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #5176 from zhouyuan/rest_bench_https

tools: Fix rest_bench to support https

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5174 from yuyuyu101/fix-bufferlist-zero

Buffer: Fix bufferlist::zero bug

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph.spec.in: snappy-devel for all supported distros

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: make /var/run/ceph conditional

/var/run/ceph should only be installed on Fedora, RH/CentOS, and
all supported SUSE distros that do not have systemd (e.g. SLE11-SP3).

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: add missing -%{release}

We have it everywhere else and I can't think of any reason why
it should be omitted here.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5179 from yghannam/yghannam-workunit-fixes-pjd

Workunits : suites/pjd.sh : Do make clean so make can build on curren…

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5178 from yghannam/yghannam-workunit-fixes-chmod

Workunits : fs/misc/chmod.sh : Include ACL characters in permission c…

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Workunits : fs/misc/chmod.sh : Include ACL characters in permission check.

Signed-off-by: Yazen Ghannam <yazen.ghannam@linaro.org>

Workunits : suites/pjd.sh : Do make clean so make can build on current arch.

Signed-off-by: Yazen Ghannam <yazen.ghannam@linaro.org>

Fix rest_bench to support https

Fixes: #3968
There's typo in the code which has compared 'http' twice

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

buffer: Fix bufferlist::zero bug with special case

Fixes: #12252
Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

UnittestBuffer: Add bufferlist zero test case

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Fix mds dump_ops_in_flight crashing ocassionally

rgw: Document the layout of pools and objects

This has to be kept in sync as the layout changes. Like any other
document, really.

Signed-off-by: Pete Zaitcev <zaitcev@redhat.com>

Merge pull request #4976 from t-miyamae/wip-mshec-fixlogic2

erasure code: fix shec's recovery issues found in teuthology test

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4813 from dachary/wip-11667-erasure-code-add-osd

tests: verify erasure code read / write after remapping

Reviewed-by: Kefu Chai <kchai@redhat.com>

Remove git build-time dependency

The git command is currently only needed in the src/make_version script.
However, we can simply avoid it by assuming that if the git command is
not installed then the current dir is not a git repo.

Signed-off-by: Boris Ranto <branto@redhat.com>

StoreTest: Add zero test for SyntheticTest

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>