mds: MDSMap: adjust buffer size for uint64 values with more than 5 chars

Fixes: #6620
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #730 from ceph/wip-monc-ping

mon: MonClient: ping monitors without authenticating

* add support on the monitor to reply to MPing messages with the contents of
  'mon_status' and 'health', regardless of a client having authenticated beforehand.

* add support on the MonClient to send a MPing message to a randomly picked
  monitor (it was easier this way, '-m ip:port' allows for targeted ping) and block
  waiting for a reply.

* add support on librados, pybind/rados.py and the 'ceph' tool to send pings to
  monitors.

Resolves: #5984

Reviewed-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

cli: ceph: add support to ping monitors

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

pybind: rados: ping a monitor via librados

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

pybind: rados: support ETIMEDOUT on make_ex()

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

librados: support pinging a monitor without auth via RadosClient

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: MonClient: allow pinging a monitor without authenticating first

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: MonClient: adjust whitespaces

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: Monitor: reply to ping messages, letting them know we're alive

Fixes: #5984
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: Monitor: do not flush formatter at end of _mon_status()

Delegate that to the caller so that we can combine the result of
_mon_status() with the result of other functions.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge remote-tracking branch 'gh/wip-6242-b' into next

Reviewed-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>

pybind/rados: create InterruptedOrTimeoutError exception

Signed-off-by: Sage Weil <sage@inktank.com>

ceph: move timeout

Signed-off-by: Sage Weil <sage@inktank.com>

ceph: catch exceptions thrown during the rados handle init

In my case, making ceph.conf unreadable triggers an exception here:

Traceback (most recent call last):
  File "./ceph", line 802, in <module>
    sys.exit(main())
  File "./ceph", line 575, in main
    conf_defaults=conf_defaults, conffile=conffile)
  File "/home/sage/src/ceph/src/pybind/rados.py", line 221, in __init__
    self.conf_read_file(conffile)
  File "/home/sage/src/ceph/src/pybind/rados.py", line 272, in conf_read_file
    raise make_ex(ret, "error calling conf_read_file")
rados.Error: error calling conf_read_file: errno EACCES

Signed-off-by: Sage Weil <sage@inktank.com>

ceph: show basic help before initializing cluster connection

That way we still get help if there is an error.

Signed-off-by: Sage Weil <sage@inktank.com>

ceph: default 5 second timeout for -h

Signed-off-by: Sage Weil <sage@inktank.com>

ceph: add --connect-timeout option

Signed-off-by: Sage Weil <sage@inktank.com>

ceph: print basic options before connecting

This will let you control-c if the mon is unresponsive.

Signed-off-by: Sage Weil <sage@inktank.com>

ceph: fixup do_help() function connection check

We might have cluster_handle defined, but not have
successfully connected. Instead, check if it's in
the connected state.

Signed-off-by: Greg Farnum <greg@inktank.com>

ceph.in: add emacs modeline

Signed-off-by: Sage Weil <sage@inktank.com>

Merge branch 'wip-6606' into next

Reviewed-by: Sage Weil <sage@inktank.com>

rgw: don't bother to call c_str() on strings passed to dump_string()

dump_string() also accepts const string reference.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #754 from ceph/wip-test-librbd

fix ceph_test_lirbd crash
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

ceph_test_librbd: fix heap overrun

We are storing rbd_snap_info_t structs, not pointers to them.  But we
can also avoid the heap entirely.

This crashed pretty reliably on arm.

Signed-off-by: Sage Weil <sage@inktank.com>

rgw: init src_bucket_name, src_object in the S3 handler

Be consistent and initialize these fields also in the S3 case.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: get rid of req_state.bucket_name

No need for this field, as we already have req_state.bucket_name_str.
This saves us some memory allocation / freeing through every request
processing.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: turn swift COPY into PUT

Fixes: #6606
The swift COPY operation is unique in a sense that it's a write
operation that has its destination not set by the URI target, but by a
different HTTP header. This is problematic as there are some hidden
assumptions in the code that the specified bucket/object in the URI is
the operation target. E.g., certain initialization functions, quota,
etc. Instead of creating a specialized code everywhere for this case
just turn it into a regular copy operation, that is, a PUT with
a specified copy source.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

librbd: parse args to ceph_test_librbd

Signed-off-by: Sage Weil <sage@inktank.com>

librbd: fix build error

From a10703008f7f4eb57a62658f2b9a015eb81509d0.

Signed-off-by: Sage Weil <sage@inktank.com>

ceph-mon: add debug to ip selection

This will hopefully help us track down #5804.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #753 from ceph/wip-5668-b

librbd: wire up flush counter
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

librbd: wire up flush counter

Fixes: #5668
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #751 from ceph/wip-6603

a couple trivial leaks

common/BackTrace: fix memory leak

Signed-off-by: Sage Weil <sage@inktank.com>

common/cmdparse: fix memory leak

demangle is allocating with malloc() in this case.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #746 from ceph/wip-6582

Wip 6582

Reviewed-by: Sage Weil <sage@inktank.com>

ReplicatedPG: copy: conditionally requeue copy ops when cancelled

We may need to requeue copy ops which are cancelled as part of an acting
set change but don't change the primary. To support this, add a
"requeue" flag to cancel_copy_ops() and copy_ops(), as well as to
CopyResults. The CopyCallback is then responsible for requeuing (the
higher layers can't do so as they can't know which request actually
triggered the copy).

Signed-off-by: Greg Farnum <greg@inktank.com>

PG: add a requeue_op() function to complement requeue_ops().

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge branch 'next'

Merge pull request #737 from xarses/6127

Add Redhat init script option
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Add Redhat init script option

Resolves: 6127
Signed-off-by: Andrew Woodward <awoodward@mirantis.com>

Merge pull request #738 from ceph/wip-cache-crc

fix cached crc, bug #6583

Reviewed-by: Samuel Just <sam.just@inktank.com>

common/buffer: invalidate crc on zero, copy_in

This does not capture users who

 - calc a crc
 - use c_str() to modify the buffer content
 - (re)calc a crc

Signed-off-by: Sage Weil <sage@inktank.com>

common/buffer: fix crc_map types

Signed-off-by: Sage Weil <sage@inktank.com>

common/buffer: drop unused fields

Signed-off-by: Sage Weil <sage@inktank.com>

qa/workunits/rest/test.py: fix mds {add,remove}_data_pool test

Arg name changed from poolid to pool in e2602c54.

Signed-off-by: Sage Weil <sage@inktank.com>

doc/release-notes: link ot the changelog

Signed-off-by: Sage Weil <sage@inktank.com>

doc/release-notes: v0.61.9

Signed-off-by: Sage Weil <sage@inktank.com>

Makefile: fix /sbin vs /usr/sbin behavior

Instead of telling configure to put things in /sbin, explicitly put the
two important items (mkcephfs and mount.fuse.ceph) in /sbin via an
automake rule.  This unbreaks FreeBSD 9.1 and probably others.

Based on patches originally from Alan Somers <asomers@gmail.com>, modified
for the current Makefile structure and applied to the specfile too.

Fixes: #6456
Signed-off-by: Sage Weil <sage@inktank.com>
Tested-by: Alan Somers <asomers@gmail.com>

OSD: check for splitting when processing recover/backfill reservations

Fixes: 6565
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #691 from ceph/wip-dirfrag

Reviewed-by: Greg Farnum <greg@inktank.com>
Partly-Reviewed-by: Sage Weil <sage@inktank.com>

v0.71

Merge remote-tracking branch 'gh/next'

.gitignore: ceph-kvstore-tool

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/next'

debian, specfile: fix ceph-kvstore-tool packaging

See da69fa09c8274585225471c68c1acc788a3881f4

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #732 from ceph/wip-kvstore-tool

tools: ceph-kvstore-tool: Access & manipulate leveldb store using the KeyValueDB interface

Reviewed-by: Sage Weil <sage@inktank.com>

ceph-kvstore-tool: copy one leveldb store to some other place

Iterates over the provided source store's keys and copies them to the
provided destination store.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

ceph-kvstore-tool: calc store crc

Reuse 'list()' function to traverse the store and calc not only version's
crcs, but also calc the store's crc.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #655 from ceph/wip-cache-crc

buffer: cache crc values in buffer::raw

Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

tools: move 'test_store_tool' to 'tools/ceph-kvstore-tool'

ceph-kvstore-tool allows for lower-level access to leveldb stores.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #709 from ceph/wip-filerecover

This patch prevents us from inadvertently reducing sparse file sizes during recovery.
We also reduce some code duplication by using eval() directly in do_file_recover()
instead of reproducing the parts we care about.

Reviewed-by: Greg Farnum <greg@inktank.com>

common/buffer: behave when cached crc stats don't start at 0

In particular, this helps when we set CEPH_BUFFER_TRACK=true

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #724 from ceph/wip-4047

cls_rbd: do not make noise in osd log on rbd removal

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

crc32c: expand unit test

Verify crc on a buffer range, and on a NULL buffer (with various
sizes).

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

sctp_crc32c: fix crc calculation in the NULL buffer case

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

crc32c: add a special case crc32c NULL buffer handling

Adding a special case crc32c NULL buffer handling for the intel assembly
code.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

common/buffer: pass NULL buffer instead of a zero-filled one

This is about 2x faster on my box using the sctp code.

Signed-off-by: Sage Weil <sage@inktank.com>

common/crc32c: document interface

Signed-off-by: Sage Weil <sage@inktank.com>

common/crc32c_intel_baseline: accept null data buffer

Signed-off-by: Sage Weil <sage@inktank.com>

common/sctp_crc32: accept NULL buffer pointer

If we get NULL we behave as if it were a zero-filled buffer.

Signed-off-by: Sage Weil <sage@inktank.com>

common/buffer: explicitly init zbuf to zeros

This might not be necessary?

Signed-off-by: Sage Weil <sage@inktank.com>

common/buffer: add crc caching performance test

On my old box:
 - matching cached values is a big win (free), obviously
 - the adjustment is the same speed as redoing the calculation.  this
   is probably because the data is already in L1/L2 cache; we still
   save memory bandwidth.

Signed-off-by: Sage Weil <sage@inktank.com>

common/buffer: instrument utilization of cached crcs

This is similar to the alloc tracking, but I've added a method to let
you enable it easily (in this case, from the unit test).

Signed-off-by: Sage Weil <sage@inktank.com>

common/buffer: cache crcs in buffer::raw

Signed-off-by: Samuel Just <sam.just@inktank.com>

include: add Spinlock

Signed-off-by: Sage Weil <sage@inktank.com>

cls_rbd: do not make noise in osd log on rbd removal

ubuntu@burnupi06:~$ tail -f /var/log/ceph/ceph-osd.1.log
2013-02-07 17:00:30.565749 7fdb09e6b700 0 <cls> cls/rbd/cls_rbd.cc:1615: error reading id for name 'sds': -2
2013-02-07 17:00:30.566301 7fdb0a66c700 0 <cls> cls/rbd/cls_rbd.cc:1521: error reading name to id mapping: -2
2013-02-07 17:03:54.085700 7fdb0a66c700 0 <cls> cls/rbd/cls_rbd.cc:1615: error reading id for name 'sdfsd': -2
2013-02-07 17:03:54.086143 7fdb09e6b700 0 <cls> cls/rbd/cls_rbd.cc:1521: error reading name to id mapping: -2

Fixes: #4047
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #728 from ceph/wip-6059

Reviewed-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
Reviewed-by: David Zafman <david.zafman@inktank.com>

Merge pull request #729 from ceph/wip-5716

rgw: gracefully handle bad root pool names

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

test_ceph_argparse: fix typo

Signed-off-by: Sage Weil <sage@inktank.com>

test_ceph_argparse: fix mds {add,remove}_data_pool tests

Signed-off-by: Sage Weil <sage@inktank.com>

common: fix non-daemon init

In commit 4f403c26dc0048ad63e20d20369fa86bfb31c50e we broke the general
non-daemon case.

Also make a note in the release notes.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

test/filestore/run_seed_to.sh: avoid obsolete --filestore-xattr-use-omap

This option no longer exists.

Signed-off-by: Sage Weil <sage@inktank.com>

MonCommands: note that pg dump options don't work in plaintext

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

rgw: gracefully handle bad root pool names

Fixes: #5716
When invalid root pools specified (don't start with a period) we return
with an error instead of asserting.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

ceph_test_rados: fix snap remove vs rollback fix

In commit 55d279b98553ba4542219b126fc7159b20b18b1f we tried to fix a race
between rollback and snap removal, but got the logic wrong: we need to
prevent *snap removal* on in-use snaps, not prevent multiple rollbacks on
the same snap.

Fixes: #6254 (again)
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

os/LevelDBStore: handle deletion race when checking store size

This fixes the fix in 64774e5792f136df2bc78db686440fc2f3a7643f which mixed
up the return value and errno.

Fixes: #6550
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

test/librados/cmd: fix compile error

From commit f0805cb61fccf9ffaa905b8477a131b4d46379fa.

Signed-off-by: Sage Weil <sage@inktank.com>

ReplicatedPG: remove the other backfill related flushes

Signed-off-by: Samuel Just <sam.just@inktank.com>

RadosModel: send racing read on write

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: block reads on an object until the write is committed

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: ping tphandle during pg removal

Fixes: #6528
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #687 from ceph/wip-pool

mon: make 'mon {add,remove}_data_pool ...' take pool name or id

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #718 from ceph/wip-6553

rgw: fix authenticated users acl group check

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #723 from ceph/wip-6334

common: don't do special things for unprivileged daemons

Reviewed-by: Greg Farnum <greg@inktank.com>

common: don't do special things for unprivileged daemons

We generally do want an admin socket and log, and people usually run these
as root.  This is more confusing than useful.

Fixes: #6334
Signed-off-by: Sage Weil <sage@inktank.com>

mon, osd: send leveldb log to /dev/null by default

Fixes: #5175
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #542 from ceph/wip-6122

test/librados/cmd.cc: tolerate thrashing on pg_command tests

doc: Update from user feedback. Needed to enable S3/Keystone.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

os/FileStore: fix fiemap double-free(s)

If GenericFileStoreBackend::do_fiemap returns anything other than 0, then
fiemap will not be allocated. However,
GenericFileStoreBackend::detect_features will free fiemap regardless,
triggering an assertion in tcmalloc. The attached patch will fix the bug
by only freeing fiemap when necessary.

In my case, do_fiemap returns -ENOSYS because fiemap is not implemented
on FreeBSD. However, this bug could also happen on Linux when do_fiemap
returns -ENOMEM.

Fixes: #6504
Signed-off-by: Alan Somers <asomers@gmail.com>
Reviewed-by: Sage Weil <sage@inktank.com>