Merge pull request #507 from ceph/wip-4635.master

Bunch of tidying up on monitor services & fix #4635

Reviewed-by: Sage Weil <sage@inktank.com>

PendingReleaseNotes: reflect 'osd crush set' behavior change

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

vstart.sh: s/osd crush set/osd crush add/ as it's supposed to be

'osd crush set' should only be used to update already existing items on
the map whereas 'osd crush add' should be able to 'add and update' items.

Considering at that point we are effectively adding a new item to the
crush map, use 'add' instead of 'set'.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: OSDMonitor: don't expose uncommitted state on 'osd crush add/set'

Fixes: #4635
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: OSDMonitor: document 'prepare_command' wrt expected behavior of no-ops

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: OSDMonitor: don't expose uncommitted state on 'osd crush link'

Fixes: #4635
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: clarify 'osd crush add' vs 'osd crush set'

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon/MonCap: remove useless 'osd crush add' perm from profile bootstrap-osd

Bootstrap doesn't use or need this; the crush update happens when the osd
starts up (see init-ceph or upstart/ceph-osd.conf).

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: AuthMonitor: fix some >80 columns debug strings

Give AuthMonitor a new look.  She sure deserves it.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: AuthMonitor: fix whitespaces

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: AuthMonitor: remove dead code

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon/PGMap: fix typo

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #501 from liewegas/wip-5906

gussy up ceph -s

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #502 from ceph/wip-mon-log-level

mon: mon_cluster_log_file_level

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #490 from dachary/master

ant is missing from the list of packages to install

Merge pull request #396 from hutkev/wip-wireshark-alt

Adding alternative Wireshark dissector.

Signed-off-by: Kevin Jones <k.j.jonez@gmail.com>

Merge pull request #499 from dachary/wip-test_osd_types

remove racy test assertions

Ceph-qa: change the fsx.sh to support hole punching test

This patch change the fsx.sh to pull better fsx.c from xfstests site
to support hole punching test.

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>
Signed-off-by: Li Wang <liwang@ubuntukylin.com>

doc: Fixed typo.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

mon/PGMap: degraded what?

Signed-off-by: Sage Weil <sage@inktank.com>

mon: status: client io, recovery io prefixes

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #500 from ceph/wip-5910

Resolves #5910.

Reviewed-by: Greg Farnum <greg@inktank.com>

OSDMonitor: add 'osd perf' command to dump recent osd perf information

Signed-off-by: Samuel Just <sam.just@inktank.com>

ObjectStore: add ObjectStore::get_cur_stats and impl in FileStore

Signed-off-by: Samuel Just <sam.just@inktank.com>

doc: Added dumpling to RPM installation.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Minor tweaks to debian install.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added dumpling installation for Debian/Ubuntu.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

remove racy test assertions

Do not assert before the loop waiting for the thread to complete the
expected side effect. The whole point of the loop is to make sure
there is no window of opportunity for a race condition and asserting
before it means taking a useless risk. If run enough times, it will
happen.

Signed-off-by: Loic Dachary <loic@dachary.org>

qa/workunits/fs/multiclient_sync_read_eof.py

Simple test to verify clients behave at EOF.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge branch 'next'

osd_types: add osd_perf_stat_t type

Signed-off-by: Samuel Just <sam.just@inktank.com>

client: add enclosing object for asok dumps

Signed-off-by: Sage Weil <sage@inktank.com>

pybind/ceph_argparse: GPL -> LGPL2

Danny Al-Gaaf <danny.al-gaaf@bisect.de>
Dan Mick <dan.mick@inktank.com>

both consented to this change via email.

Signed-off-by: Sage Weil <sage@inktank.com>

rbd.cc: relicense as LGPL2

All past authors for rbd.cc have consented to relicensing from GPL to
LGPL2 via email:

---

Date: Sat, 27 Jul 2013 01:59:36 +0200
From: Sylvain Munaut <s.munaut@whatever-company.com>
Subject: Re: Ceph rbd.cc GPL -> LGPL2 license change

I hereby consent to the relicensing of any contribution I made to the
aforementioned rbd.cc file from GPL to LGPL2.1.

(I hope that'll be impressive enough, I did my best :p)

btw, tnt@246tNt.com and s.munaut@whatever-company.com are both me.

Cheers,

    Sylvain

---

Date: Fri, 26 Jul 2013 17:00:48 -0700
From: Yehuda Sadeh <yehuda@inktank.com>
Subject: Re: Ceph rbd.cc GPL -> LGPL2 license change

I consent.

---

Date: Fri, 26 Jul 2013 17:02:24 -0700
From: Josh Durgin <josh.durgin@inktank.com>
Subject: Re: Ceph rbd.cc GPL -> LGPL2 license change

I consent.

---

Date: Fri, 26 Jul 2013 18:17:46 -0700
From: Stanislav Sedov <stas@freebsd.org>
Subject: Re: Ceph rbd.cc GPL -> LGPL2 license change

I consent.

Thanks for taking care of it!

---

Date: Fri, 26 Jul 2013 18:24:15 -0700
From: Colin McCabe <cmccabe@alumni.cmu.edu>

I consent.

cheers,
Colin

---

Date: Sat, 27 Jul 2013 07:08:12 +0200
From: Christian Brunner <christian@brunner-muc.de>
Subject: Re: Ceph rbd.cc GPL -> LGPL2 license change

I consent

Christian

---

Date: Sat, 27 Jul 2013 12:17:34 +0300
From: Stratos Psomadakis <psomas@grnet.gr>
Subject: Re: Ceph rbd.cc GPL -> LGPL2 license change

Hi,

I consent with the GPL -> LGL2.1 re-licensing.

Thanks
Stratos

---

Date: Sat, 27 Jul 2013 16:13:13 +0200
From: Wido den Hollander <wido@42on.com>
Subject: Re: Ceph rbd.cc GPL -> LGPL2 license change

I consent!

You have my permission to re-license the code I wrote for rbd.cc to LGPL2.1

---

Date: Sun, 11 Aug 2013 10:40:32 +0200
From: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
Subject: Re: btw

Hi Sage,

I agree to switch the license of ceph_argparse.py and rbd.cc from GPL2
to LGPL2.

Regards

Danny Al-Gaaf

---

Date: Tue, 13 Aug 2013 17:15:24 -0700
From: Dan Mick <dan.mick@inktank.com>
Subject: Re: Ceph rbd.cc GPL -> LGPL2 license change

I consent to relicense any contributed code that I wrote under LGPL2.1 license.

---

...and I consent too.  Drop the exception from COPYING and debian/copyright
files.

Signed-off-by: Sage Weil <sage@inktank.com>

mon/PGMap: make pg state indentation cleaner

Signed-off-by: Sage Weil <sage@inktank.com>

mon: status: only include mdsmap if epoch > 1

Signed-off-by: Sage Weil <sage@inktank.com>

mon: move recovery rate to a separate line

Signed-off-by: Sage Weil <sage@inktank.com>

mon: move mdsmap up

Signed-off-by: Sage Weil <sage@inktank.com>

mon/PGMap: make si modifiers look prettier

Signed-off-by: Sage Weil <sage@inktank.com>

mon: make pg info in 'status' more informative and visually parseable

Signed-off-by: Sage Weil <sage@inktank.com>

types: pretty_si_t

Similar to si_t, but leaves a space between the numbers and the units.  In
the degenerate case (no M, K, etc. modifier) there's simply a trailing
space.  For example,

  out << pretty_si_t(num) << "objects/sec";

will look pretty.

Signed-off-by: Sage Weil <sage@inktank.com>

mon/PGMap: break out print_oneline_summary

Signed-off-by: Sage Weil <sage@inktank.com>

mon: expand osd summary (flags on separate line)

Signed-off-by: Sage Weil <sage@inktank.com>

osdmap: move oneliner summary to separate function

We're about to make print_summary() do something much nicer.

Signed-off-by: Sage Weil <sage@inktank.com>

librados: fix MWatchNotify leak

Do not leak the message if the watcher is not registered.  Also, simplify
this block.

Fixes (part of): #5949
Backport: dumpling, cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: do not leak handler in get_handler() error path

If we fail to initialize, delete the handler.

Fixes (part of): #5949
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

v0.67

Merge pull request #497 from ksperis/rbdmap.init-3

Rbdmap.init 3

Reviewed-by: Sage Weil <sage@inktank.com>

doc/release-notes: fix rst

Signed-off-by: Sage Weil <sage@inktank.com>

doc/release-notes: upgrade sequence

Signed-off-by: Sage Weil <sage@inktank.com>

doc/release-notes: roll-up of upgrade/compat notes from cuttlefish to dumpling

Signed-off-by: Sage Weil <sage@inktank.com>

init-rbdmap: minor fix no rbd exist

Avoid error if stop when no rbd device exist

Signed-off-by: Laurent Barbe <laurent@ksperis.com>

init-rbdmap: fix for recursive umount

Umount is not always done in the correct order.
For exemple in that case :
/dev/rbd1 on /myrbd
/dev/rbd2 on /myrbd/.snapshots/@GMT-2013.08.09-10.14.44
rbd2 should be umounted before rbd1

Signed-off-by: Laurent Barbe <laurent@ksperis.com>

Makefile.am: fix libglobal.la races

Several targets had libglobal.la in the  _LDFLAGS macro definition
when it should have been in the _LDADD macro.  Remove those occurrance
and add the LIBGLOBAL_LDA macro to the targets _LDADD instead.

See related commit 71f3e56d4b2c07f5fc56522a2080f1ff101ef340

Signed-off-by: Gary Lowell <gary.lowell@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>

rgw: fix multi delete

Fixes: #5931
Backport: bobtail, cuttlefish

Fix a bad check, where we compare the wrong field. Instead of
comparing the ret code to 0, we compare the string value to 0
which generates implicit casting, hence the crash.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

mon: mon_cluster_log_file_level

By popular demand, control the min level of logging that goes to
/var/lib/ceph/ceph.log.

Signed-off-by: Sage Weil <sage@inktank.com>

ceph-disk: fix mount options passed to move_mount

Commit 6cbe0f021f62b3ebd5f68fcc01a12fde6f08cff5 added a mount_options but
in certain cases it may be blank.  Fill in with the defaults, just as we
do in mount().

Backport: cuttlefish
Reviewed-by: Dan Mick <dan.mick@inktank.com>
Signed-off-by: Sage Weil <sage@inktank.com>

config_opts.h: reduce osd_recovery_max_active and osd_recovery_max_single_start

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: David Zafman <david.zafman@inktank.com>

doc/release-notes: adjust whitespace

Signed-off-by: Sage Weil <sage@inktank.com>

doc/release-notes: v0.67 release notes

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/next'

ceph: retry new-style mon command if we get EINVAL from the old-style command

We can race with a mon upgrade:

- get command descriptions, get EINVAL
- mons upgrade, new quorum
- send old-style command
- get EINVAL

In this case, we should try one last time to get the command descriptions.

Fixes: #5788
Reviewed-by: Dan Mick <dan.mick@inktank.com>
Signed-off-by: Sage Weil <sage@inktank.com>

common: pick_addresses: fix bug with observer class that triggered #5205

The Observer class we defined to observe conf changes and thus avoid
triggering #5205 (as fixed by eb86eebe1ba42f04b46f7c3e3419b83eb6fe7f9a),
was returning always the same const static array, which would lead us to
always populate the observer's list with an observer for 'public_addr'.

This would of course become a problem when trying to obtain the observer
for 'cluster_add' during md_config_t::set_val() -- thus triggering the
same assert as initially reported on #5205.

Backport: cuttlefish
Fixes: #5205
Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #486 from kri5/wip-s3-compliance-doc

Wip s3 compliance doc

Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: return 423 Locked response when failing to lock object

Fixes: #5882
Translate the EBUSY we get when trying to lock a shard / object
to 423 Locked response. Beforehand it was just translated to the
default 500.

Reviewed-by: Sage Weil <sage@inktank.com>
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: make RGWHTTPClient callback methods pure virtual

This will make it easier to catch interface changes, like the
ones that triggered #5921.

Reviewed-by: Sage Weil <sage@inktank.com>
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: rename read_header() to receive_header() where needed

The same as with #5921, fixing a missed interface change.

Reviewed-by: Sage Weil <sage@inktank.com>
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: rename data receive callbacks in swift token revocation

Fixes: #5921
As part of the work that was made for dumpling, the http
client in-data callback was renamed in order to avoid confusion.
However, we missed the rename in a couple of places, which this
patch amend.

Reported-by: Roald van Loon <roaldvanloon@gmail.com>
Reviewed-by: Sage Weil <sage@inktank.com>
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge remote-tracking branch 'gh/wip-5648-c'

Reviewed-by: Sage Weil <sage@inktank.com>

PendingReleaseNotes: note 'ceph daemon ...' argument behavior change

Signed-off-by: Sage Weil <sage@inktank.com>

doc: Add a page about rgw S3 API compliance

Signed-off-by: Christophe Courtaut <christophe.courtaut@gmail.com>

ant is missing from the list of packages to install

to build the documentation.

Signed-off-by: Loic Dachary <loic@dachary.org>

RadosClient: shutdown monclient after dropping lock

Otherwise, the monclient shutdown may deadlock waiting
on a context trying to take the RadosClient lock.

Fixes: #5897
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

mds: notify clients about deleted inode

To make sure clients trim the deleted inode from the their cache
ASAP. After all clients release the inode, we can reclaim space.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'yan/wip-mds'

Reviewed-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/next'

buffer: change #include order

huangjun <hjwsm1989@gmail.com> reports that this fixes the compilation on
CentOS5.

Signed-off-by: Sage Weil <sage@inktank.com>

make sure we are using the mount options

Signed-off-by: Alfredo Deza <alfredo@deza.pe>
(cherry picked from commit 34831d0989d4bcec4920068b6ee09ab6b3234c91)

Merge pull request #480 from dachary/master

Add the definition of PG temp to the glossary

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #487 from ceph/wip-5746

ceph.in: Re-enable ceph interactive mode (missing its output).

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #489 from ceph/wip-5903

make sure we are using the mount options

Backport: dumpling, cuttlefish
Reviewed-by: Sage Weil <sage@inktank.com>

make sure we are using the mount options

Signed-off-by: Alfredo Deza <alfredo@deza.pe>

Merge pull request #488 from dmick/wip-dmick-daemon-einval

Wip dmick daemon einval

Reviewed-by: Sage Weil <sage@inktank.com>

ceph.in: return EINVAL on daemon command error

Signed-off-by: Dan Mick <dan.mick@inktank.com>

ceph.in: Re-enable ceph interactive mode (missing its output).

Also, loop on error.  There's no reason to exit the interpreter loop on
an error, and it's probably less annoying if we don't.  Print the error,
and any output, and continue.

Fixes: #5746
Signed-off-by: Dan Mick <dan.mick@inktank.com>

mon: fix 'osd crush rule rm ...' dup arg

This was broken way back in 0d66c9ebbf626117c641c975a8682a0aaba588c4, but
we were ignoring the dup until recently.
t
Signed-off-by: Sage Weil <sage@inktank.com>

qa/workunits/cephtool/test.sh: test set/unset of all osd flags

Signed-off-by: Sage Weil <sage@inktank.com>

mon/MonCommands: fix typo nobackfile -> nobackfill

Signed-off-by: Sage Weil <sage@inktank.com>

Add back the mistakenly removed "ceph osd set noscrub/nodeep-scrub"

Fixes: #5868
Signed-off-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

Merge branch 'wip-recovery-op-warn' into next

Reviewed-by: Sage Weil <sage@inktank.com>

OSD: suspend timeout on process_peering_events pg lock

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #476 from dalgaaf/wip-CID-1058791

Fix CID 1058791

Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

test: mon: moncap: add 'allow command foo with arg="bar.baz"' tests

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa: workunits: mon: workunit to stress the monitor's caps

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: AuthMonitor: change 'auth add' behaviour

If an entity already existed, 'auth add' would smash its key and caps
with whatever was on the supplied keyring file; if a keyring weren't
specified, we would simply generate a new key and destroy all existing
caps (unless caps were specified and happened to be different from the
already in-place caps).  This behaviour is obviously sketchy.

With this patch we now enforce the following behaviour:

- if entity does not exist in current state, check if we are about to
  create it (by checking the pending state); if so, wait for the new state
  to be committed and re-handle the command then, so we don't get bad
  results from pending request
- if the command reproduces the current state (same key, same caps), we
  return 0; else,
- if entity exists and supplied key OR caps are different, return -EINVAL
- else create a new entity.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

mon: drop old _allowed_command()

Signed-off-by: Sage Weil <sage@inktank.com>

mon: services: no longer needed to enforce caps on a per-service basis

We now perform all perm checks for commands on Monitor::handle_command().
Services no longer need to check them.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: Monitor: purge old permission checking format on handle_command()

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: Monitor: check caps considering command's requirements

Fixes: #5648
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

OSD: only start osd_recovery_max_single_start at once

Signed-off-by: Samuel Just <sam.just@inktank.com>