Merge pull request #5324 from ceph/wip-12355

mds: fix crash while stopping rank

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mds: fix crash while stopping rank

trim() was broken in:
8a91daae mds: fix mds crash when mds_max_log_events smaller

When doing a final trim in shutdown we need to trim everything,
not just all but the last segment.  The previous change's intent
was to prevent badness when mds_log_max_events was too small: accomplish
the same thing here by clamping the value to something sane.

Fixes: #12355
Signed-off-by: John Spray <john.spray@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge branch 'wip-mon-scrub'

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-11786'

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-11470'

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mon: PaxosService: call post_refresh() instead of post_paxos_update()

Whenever the monitor finishes committing a proposal, we call
Monitor::refresh_from_paxos() to nudge the services to refresh.  Once
all services have refreshed, we would then call each services
post_paxos_update().

However, due to an unfortunate, non-critical bug, some services (mainly
the LogMonitor) could have messages pending in their
'waiting_for_finished_proposal' callback queue [1], and we need to nudge
those callbacks.

This patch adds a new step during the refresh phase: instead of calling
directly the service's post_paxos_update(), we introduce a
PaxosService::post_refresh() which will call the services
post_paxos_update() function first and then nudge those callbacks when
appropriate.

[1] - Given the monitor will send MLog messages to itself, and given the
service is not readable before its initial state is proposed and
committed, some of the initial MLog's would be stuck waiting for the
proposal to finish.  However, by design, we only nudge those message's
callbacks when an election finishes or, if the leader, when the proposal
finishes.  On peons, however, we would only nudge those callbacks if an
election happened to be triggered, hence the need for an alternate path
to retry any message waiting for the initial proposal to finish.

Fixes: #11470
Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge branch 'wip-11545'

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5251 from ceph/wip-12021-next

OSDMonitor: allow addition of cache pool with non-empty snaps with co…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5188 from athanatos/wip-11687

PG::find_best_info: ignore info.les for incomplete peer

Reviewed-by: Sage Weil <sage@redhat.com>

mon: Monitor: use 'ceph mon metadata' instead of 'ceph mon_metadata'

'ceph mon_metadata' was added still during this dev cycle, so there is
no need to deprecate it first.

Fixes: #11545
Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #5238 from tweenk/out-of-tree-build-fix

Makefile: fix out-of-tree build.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5256 from ceph/wip-ftbfs-i386-cls-cephfs

fix the FTBFS on i386 due to libcls_cephfs

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #5255 from hjwsm1989/fix-pgmonitor

mon/PGMonitor: use poolname reference instead of get it in osdmap

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: Monitor: use 'ceph mon sync force' instead of 'ceph sync force'

Makes it easier to identify the command as being related with the
monitor instead of cluster-wide.

This entails adding an exception to module 'mon' in order to have this
command handled by the Monitor class instead of MonmapMonitor (which is
the one traditionally handling 'mon' module commands).

Fixes: #11545
Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: Monitor: use 'ceph mon scrub' instead of 'ceph scrub'

Makes it easier to identify the command as being related with the
monitor instead of cluster-wide.

This entails adding an exception to module 'mon' in order to have this
command handled by the Monitor class instead of MonmapMonitor (which is
the one traditionally handling 'mon' module commands).

Fixes: #11545
Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: Monitor: use 'ceph mon compact' instead of 'ceph compact'

Makes it easier to identify the command as being related with the
monitor instead of cluster-wide.

This entails adding an exception to module 'mon' in order to have this
command handled by the Monitor class instead of MonmapMonitor (which is
the one traditionally handling 'mon' module commands).

Fixes: #11545
Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa/workunits: cephtool: test deprecated commands

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

PendingReleaseNotes: note deprecation of commands

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: MonCommands.h: DEPRECATE 'ceph sync force'

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: MonCommands.h: DEPRECATE 'ceph compact'

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: MonCommands.h: DEPRECATE 'ceph scrub'

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: Monitor: allow deprecating commands and debugging as obsolete

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: Monitor: add is_noforward() helper to MonCommand

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: Monitor: add support to have OBSOLETE commands

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: MonCommand: don't match help string in is_compat()

Otherwise it's virtually impossible to change a command's help string
without triggering a mismatch with the leader's command set.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: MonCommands: add NOFORWARD to 'ceph sync force'

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: MonCommand: add FLAG_NONE

Instead of passing '0' for commands without flags, pass FLAG_NONE
instead.  It's prettier and more obvious -- and it only costs 64 bits.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: MonCommand: have flags as uint32_t instead of enum

Makes much more sense if we're OR'ing flags.  Or not as weird.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: MonCommands: accept FLAG(f) instead of 'f' in command sig

This allows us to do nifty stuff like 'FLAG(foo) | FLAG(bar)' and expand
it to (MonCommand::FLAG_foo | MonCommand::FLAG_bar), instead of being
bound by a single flag on macro expansion.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

automake: Fix out-of-tree build.

Corrects minor Automake errors which prevented Ceph from building
when configure was invoked from a different directory than
the toplevel source directory.

Signed-off-by: Krzysztof Kosiński <krzysztof.kosinski@intel.com>

Merge pull request #5143 from trociny/fix-mds_metadata

mon: fix mds metadata

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

packaging: package libcls_cephfs.so

Signed-off-by: Kefu Chai <kchai@redhat.com>

cls: fix the build on i386

time_t is more a arithmetic type. and it's not portable. it's always
defined as "long int" by libc. and we have no encode(int, bl), which
is expected. so a safe way is to use int64_t for presenting the mtime
returned from the stat() call.

Signed-off-by: Kefu Chai <kchai@redhat.com>

  mon/PGMonitor: use poolname reference instead of get it in osdmap
Signed-off-by: huangjun <hjwsm1989@gmail.com>

Merge pull request #4917 from hjwsm1989/wip-rest-bench

 tools/rest_bench.cc: bucket is not mandatory bc we have a default one

Reviewed-by: Kefu Chai <kchai@redhat.com>

  rest_bench: bucketname is not mandatory as we have a default name

Signed-off-by: huangjun <hjwsm1989@gmail.com>

  rest_bench: drain the work queue to fix a crash
Fixes: #3896
Signed-off-by: huangjun <hjwsm1989@gmail.com>

PG::find_best_info: ignore info.les for incomplete peer

See included update to doc/dev/osd_internals/last_epoch_started.rst

Fixes: 11687
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #5249 from tchaikov/wip-11101

ceph.in: improve the error message

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #5110 from ceph/wip-11864-rgw-mime-types-pkg

11864: packaging: RGW depends on /etc/mime.types

Reviewed-by: Boris Ranto <branto@redhat.com>

packaging: RGW depends on /etc/mime.types

If the mimecap RPM or mime-support DEB is not installed, then the
/etc/mime.types file is not present on the system. RGW attempts to read
this file during startup, and if the file is not present, RGW logs an
error:

  ext_mime_map_init(): failed to open file=/etc/mime.types ret=-2

Make the radosgw package depend on the mailcap/mime-support packages so
that /etc/mime.types is always available on RGW systems.

http://tracker.ceph.com/issues/11864 Fixes: #11864

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #4699 from joehandzik/master

FileStore: Collect device partition information

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4876 from ceph/wip-11855

crush: fix 'ceph osd crush reweight-subtree'

Reviewed-by: Li Wang <liwang@ubuntukylin.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph.in: print more detailed warning for 'ceph <type> tell'

put the full signature of "tell <target> <command> [options...]"
instead of "tell {0}.<id>", which could be misleading somehow.

Fixes: 11101
Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.in: print more detailed error message for 'tell' command

* we do not allow user specify a certain daemon when starting an
  interactive session. the existing error message could lead to
  some confusion. so put more details in it.

Fixes: #11101
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5245 from yuyuyu101/wip-async-fix-18

AsyncConnection: Fix incorrect sign_message usages

Merge pull request #5244 from zhouyuan/rgw_obj_stripe_size

rgw: doc: Fix radosgw stripe size config ref

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #5240 from dachary/wip-ceph-release-notes

tools: src/script/ceph-release-notes normalization

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #5165 from zhouyuan/librados_default_crush_ruleset

Make librados pool_create respect default_crush_ruleset

Reviewed-by: Kefu Chai <kchai@redhat.com>

librados: Make librados pool_create respect default_crush_ruleset

Make pool_create in librados use the osd_pool_default_crush_replicated_ruleset

Fixes: #11640
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

9.0.2

AsyncConnection: Make sign_message ahead of construct message bufferlist

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

AsyncConnection: Fix non-fastdispatch message doesn't prepare case

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

rgw: doc: Fix radosgw stripe size config ref

s/rgw object stripe size/rgw obj stripe size
https://github.com/ceph/ceph/blob/master/src/common/config_opts.h#L1024

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: src/script/ceph-release-notes normalization

Issue an error if for the title does not start with a known prefix.
Issue an error if no issue is associated to a pull request.

Group the pull requests fixing the same issue together.

Add link to the issue and the pull request.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #5230 from majianpeng/check_ops_in_flight_cleanup

common: clean up code for OpTracker::check_ops_in_flight.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

common: clean up code for OpTracker::check_ops_in_flight.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #5220 from vumrao/wip-vumrao-12318

doc: change tcp rcvbuf and tcp nodelay to ms tcp rcvbuf and ms tcp nodelay

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: change tcp rcvbuf and tcp nodelay to ms tcp rcvbuf
     and ms tcp nodelay

Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>

Merge pull request #4939 from ceph/wip-offline-backward

cephfs-data-scan (offline backward file recovery)

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5190 from guce/master

radosgw-admin: after subuser modify print only once user info

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5183 from zhouyuan/rgw_max_put_size_configurable

rgw: Make RGW_MAX_PUT_SIZE configurable

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5213 from wido/wip-rgw-keepalive

rgw: If the client sends a Connection: close header respond accordingly.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5210 from tweenk/docs-improvements

Document librbd::parent_spec and librbd::parent_info.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5215 from SUSE/wip-fdupes

ceph.spec.in: do not run fdupes, even on SLE/openSUSE

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ceph.spec.in: do not run fdupes, even on SLE/openSUSE

In openSUSE there is a policy to use %fdupes in the spec file if RPMLINT
complains about duplicate files wasting space in the filesystem.

However, RPMLINT is not so complaining, so drop fdupes.

http://tracker.ceph.com/issues/12301 Fixes: #12301

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5102 from ceph/wip-mds-refactor-small

Minor MDS:: refactor

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5209 from zhouyuan/rgw_swift_user

rgw: Make vstart.sh print out swift user info

Reviewed-by: Sage Weil <sage@redhat.com>

packaging: add cephfs-data-scan

Signed-off-by: John Spray <john.spray@redhat.com>

tools/cephfs: add cephfs-data-scan

For scraping cephfs data pools for metadata in
backtraces, and injecting that metadata into
the cephfs metadata pool (initially offline).

Signed-off-by: John Spray <john.spray@redhat.com>

doc: add some docs about cephfs-data-scan

These are deliberately fairly sparse, because:
 * These tools are for experts
 * These tools may well be wrapped in a higher
   level recovery tool that orchestrates parallel
   workers at some stage.

Signed-off-by: John Spray <john.spray@redhat.com>

include/ceph_fs.h: define magic LOST+FOUND ino

So that recovery tools can create this dir without
having to consult/modify InoTable.

Signed-off-by: John Spray <john.spray@redhat.com>

cls: add CephFS object class

For use in cephfs-data-scan for file size and mtime
recovery, by doing a set-if-greater for size/mtime
xattrs on the 0th object of a file.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: include damage_flags in inter-mds encoding

...and correspondingly bump the cluster protocol
version as these structures are not versioned
themselves.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: bump required version on fnode_t encoding

So that older MDS instances can't miss/ignore
the damage flags.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add damage_flags_t to inode+frag

Used to mark inodes+fnodes with flags indicating
that particular types of damage have either
been detected (scrub/fetch) or caused (cephfs-data-scan
inject).

This should serve as a cue to the MDS to do online
repair of stats for the nodes that have been injected.

Signed-off-by: John Spray <john.spray@redhat.com>

mds/CInode: move hash_dentry_name up into InodeStore

Because it only depends on the data in the store, not
anything from a live MDS, and it's useful when
we're dealing with fragmented directories from
an offline tool.

Signed-off-by: John Spray <john.spray@redhat.com>

tools: fix journal reset error handling

When we failed and prompted for --force, it still
looked like a success.  ::reset never had any
proper error handling since it was lifted out
of the MDS, so add it now.

Signed-off-by: John Spray <john.spray@redhat.com>

client: fix typo

s/bye/byte/

Signed-off-by: John Spray <john.spray@redhat.com>

mds: correct typo in log message

s/issueing/issuing/

Silly, but it annoyed me every time I saw it...

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #5164 from ceph/wip-12088

client: reference counting 'struct Fh'

Reviewed-by: John Spray <john.spray@redhat.com>

rgw: If the client sends a Connection: close header respond accordingly.

HTTP/1.1 assumes Keep-Alive by default, but if a Connection: close header is send
the server should respond with it as well.

This makes the client close the connection after the request.

Fixes: #12298

Merge pull request #5201 from ceph/wip-12272

tools: Fix rados export to use io_ctx in the right way

Reviewed-by: John Spray <john.spray@redhat.com>

rgw: Make vstart.sh print out swift user info

Currently vstart.sh only print out S3 user info. This patch updates it
to print swift user info too. Also changes the swift account/user/password
to test/tester/testing which align with openstack-swift better:
https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L270

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Fixes: #12286 radosgw-admin: after subuser modify print only once user info.
remove rgw_admin.cc OPT_SUBUSER_MODIFY, show_user_info code block.

switch (opt_cmd) {
...
case OPT_SUBUSER_MODIFY:
show_user_info(info, formatter);       //show first time (remove this)
break;
...
}

// output the result of a user operation
if (output_user_info) {
...
show_user_info(info, formatter);       //show second time
}

test fix:
before: after subuser modify print twice user info.
after changes, do the same procedure, print only once user info.

Signed-off-by: guce guce@h3c.com

rgw: Make RGW_MAX_PUT_SIZE configurable

The 5GB limit of a single operation uploading was part of S3 spec.
However some private setups may have some special requirements
on this limit. It's more convinent to have a configurable value.

Fixes: #6999
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: Fix rados export to use io_ctx in the right way

Cause by: 868f6b7626db4a802b38d1d8ce5135108b322f74

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #5197 from ceph/wip-cot-meta

Wip cot meta

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5198 from tchaikov/wip-doc-erasure-fix

doc/erasure-code: fix couple typos

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/erasure-code: fix couple typos

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools, test: Some ceph-objectstore-tool error handling fixes

Improve various error messages generated with invalid syntax
Add test cases for most of these error messages

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add debug argument to the ceph-objectstore-tool test

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Check for valid --op earlier so we can get a better error message

Signed-off-by: David Zafman <dzafman@redhat.com>

tools, test: Add ceph-objectstore-tool to operate on the meta collection

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Fix newlines in output of --op list

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Fix dump-super which doesn't require pgid

A valid pgid had to be specified in order for superblock to be displayed,
but it was otherwise ignored.
Fix message that didn't include dump-super

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Check and specify commands that require the pgid specification

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #4404 from rzarzynski/wip-11431

rgw: fix lack of account name attribute in XML-formatted bucket listing of Swift account

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5078 from rzarzynski/wip-12158

rgw: send Content-Length in response for GET on Swift account.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>