erasure-code: fix assert overflow

If the file size does not fit in 32 bits the (unsigned) cast will
overflow. Cast to uint64_t which is the type of the value returned by
get_total_chunk_size.

http://tracker.ceph.com/issues/9537 Fixes: #9537

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2521 from athanatos/wip-9453

osd_types: fix pg_log_t tail dump

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2519 from athanatos/wip-9482

PG::find_best_info: let history.last_epoch_started provide a lower bound

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2520 from athanatos/wip-9497

PG::choose_acting: let the pg go down if acting is smaller than min_size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2518 from athanatos/wip-9481

PGLog::claim_log_and_clear_rollback_info: fix rollback_info_trimmed_to

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2522 from athanatos/wip-8315

ReplicatedPG: cancel cb on blacklisted watcher

Reviewed-by: Sage Weil <sage@redhat.com>

mds: fix not journaling client metadata

Previously the code was there for storing in
the SessionMap table, but not for the ESession
logevent.

Fixes: #9518
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 1395275e0f4d12631b1e4074598a7682810f00f9)

Merge pull request #2498 from ceph/wip-objecter-cancel

osdc/Objecter: cancel timeout before clearing op->session

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #2513 from ceph/wip-decode-dump

msg: dump messages we can' decode at a low devel level

Reviewed-by: Greg Farnum <greg@inktank.com>

msg: hexdump bad messages at debug level 1 (not 30)

These are rare, and when they do happen, we want to see them in the
logs.

Signed-off-by: Sage Weil <sage@redhat.com>

msg: hexdump messages we can't decode at configurable debug level

We had this hard-coded to 30.  Make it tunable.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2512 from dachary/wip-9490-crushtool-num-rep

Revert "crushtool: safeguard for missing --num-rep when --test"

Reviewed-by: John Spray <john.spray@redhat.com>

Revert "crushtool: safeguard for missing --num-rep when --test"

This reverts commit 10b8966c8da38f8ae5ac7dbb2e16a5e45e502ba3.

Merge pull request #2496 from ceph/wip-9189

#9375 and #9189 - add client metadata and improve health messages

Reviewed-by: Greg Farnum greg@inktank.com

mds: use new Session::human_name in health msgs

Signed-off-by: John Spray <john.spray@redhat.com>

mds: implement const SessionMap::get_session

Signed-off-by: John Spray <john.spray@redhat.com>

mds: generate friendly name for client sessions

...from the client metadata, for use in health messages.

Signed-off-by: John Spray <john.spray@redhat.com>

ceph_fuse: include mount point in client metadata

Signed-off-by: John Spray <john.spray@redhat.com>

client: send metadata in session open

Populated with hostname and entity_id by
default, with interface for outer layers
like ceph_fuse, libcephfs to inject their
own metadata.

Signed-off-by: John Spray <john.spray@redhat.com>

ceph_fuse: refactor init to make mount_point accessible

...so that we can read out the mount point between init
and start() in order to feed it to Client before it connects
to the MDS.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: include client metadata in `session ls`

Signed-off-by: John Spray <john.spray@redhat.com>

mds: store client metadata on Session

Signed-off-by: John Spray <john.spray@redhat.com>

messages: add cephfs client metadata to MClientSession

Signed-off-by: John Spray <john.spray@redhat.com>

mds: collapse numerous client warnings into one

...to avoid sending O(N_clients) sized beacons.

Fixes: #9375
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #2508 from dachary/wip-9490-crushtool-num-rep

crushtool: safeguard for missing --num-rep when --test

Reviewed-by: Johnu George <johnugeorge109@gmail.com>

PGLog::claim_log_and_clear_rollback_info: fix rollback_info_trimmed_to

We have been setting it to the old head value.  This is usually
harmless since the new head will virtually always be ahead of the
old head for claim_log_and_clear_rollback_info, but can cause trouble
in some edge cases.

Fixes: #9481
Backport: firefly
Signed-off-by: Samuel Just <sam.just@inktank.com>

PG::find_best_info: let history.last_epoch_started provide a lower bound

If we find a info.history.last_epoch_started above any
info.last_epoch_started, we must be missing updates and
min_last_update_acceptable should provisionally be max().

Fixes: #9482
Backport: firefly
Signed-off-by: Samuel Just <sam.just@inktank.com>

PG::choose_acting: let the pg go down if acting is smaller than min_size

Even if the backfill peer would bring us up to min_size, we can't go
active since build_prior will not consider the interval maybe_went_rw.

Fixes: #9497
Backport: firefly
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2507 from ceph/wip-9435

#9435 - enforce rules for using EC/tiered pools with CephFS

Reviewed-by: Greg Farnum <greg@inktank.com>

mon: forbid tier changes when in use by FS

* Removing tiers from a base pool in use by CephFS is forbidden.
* Using CephFS pools as tiers is forbidden.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: prevent cache pools being used CephFS

Fixes two things:
 * EC pools are now permissible if they have a cache overlay
 * Pools are not permissible if they are a cache tier.

Fixes: #9435
Signed-off-by: John Spray <john.spray@redhat.com>

crushtool: safeguard for missing --num-rep when --test

http://tracker.ceph.com/issues/9490 Fixes: #9490

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2497 from ceph/wip-xfs-inode64

ceph-disk: mount xfs with inode64 by default

Reviewed-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #2499 from ceph/wip-9219-giant

wip-9219: subscribe to the newest osdmap when reconnecting to a monitor

Reviewed-by: Sage Weil <sage@redhat.com>

osd: subscribe to the newest osdmap when reconnecting to a monitor

This is mostly relevant in testing clusters, but it ensures that an OSD
disconnecting from the monitor at the wrong time will still see any recent
map updates and prevent accidental loss of map injection into the OSD cluster.
Fixes: #9219
Signed-off-by: Greg Farnum <greg@inktank.com>

osdc/Objecter: fix command op cancellation race

Cancel the command op timeout event before we clear out the op from the
session struct.  This isn't strictly necessary because command_op_cancel
will "gracefully" handle the case where the tid is no longer present, but
this avoids that noise and is cleaner.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: cancel timeout before clearing op->session

The C_CancelOp path assumes op->session != NULL.  Cancel that op before
we clear it.  This fixes a crash like

#0  pthread_rwlock_wrlock () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_wrlock.S:39
#1  0x00007fc82690a4b1 in RWLock::get_write (this=0x18, lockdep=<optimized out>) at ./common/RWLock.h:88
#2  0x00007fc8268f4d79 in Objecter::op_cancel (this=0x1f61830, s=0x0, tid=0, r=-110) at osdc/Objecter.cc:1850
#3  0x00007fc8268ba449 in Context::complete (this=0x1f68c20, r=<optimized out>) at ./include/Context.h:64
#4  0x00007fc8269769aa in RWTimer::timer_thread (this=0x1f61950) at common/Timer.cc:268
#5  0x00007fc82697a85d in RWTimerThread::entry (this=<optimized out>) at common/Timer.cc:200
#6  0x00007fc82651ce9a in start_thread (arg=0x7fc7e3fff700) at pthread_create.c:308

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-disk: mount xfs with inode64 by default

We did this forever ago with mkcephfs, but ceph-disk didn't.  Note that for
modern XFS this option is obsolete, but for older kernels it was not the
default.

Backport: firefly
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2492 from ceph/wip-9284

#9284 - fix client RECALL handling and add health metrics

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #2476 from ceph/wip-9307

rgw: push hash calculater deeper

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2493 from ceph/wip-rbd-objectcacher-hang

rbd: ObjectCacher reads can hang when reading sparse files

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2495 from dachary/wip-erasure-code-preload

erasure-code: preload fails if < 0

Reviewed-by: Sage Weil <sage@redhat.com>

erasure-code: preload fails if < 0

And not if < -1.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2486 from jgalvez/master

init-radosgw.sysv: Support systemd for starting the gateway

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2472 from dachary/wip-9429-bench

erasure-code: fix erasure_code_benchmark goop (decode)

Reviewed-by: Janne Grunau <j@jannau.net>

mds: limit number of caps inspected in caps_tick

This is to avoid hitting an O(caps) loop in the worst
cast scenario.  This mechanism is a little crude but
should be superceded at some point by admin socket
functionality to inspect session caps so that we
don't need to spit out this level of detail in logs.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: keep per-client revoking caps list

...to avoid doing an O(caps) scan to find out
which clients are responsible for any late-revoking
caps during health checks.

Signed-off-by: John Spray <john.spray@redhat.com>

xlist: implement copy constructor

...so that I can have a std::map of them.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: health metric for late releasing caps

Follow up on Yan Zheng's "mds: warn clients which
aren't revoking cap" to include a health metric
for this condition as well as the clog messages.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: trigger transaction on MDS health changes

I think this was previously only working as a side effect
of other MDS map changes.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add a health metric for failure to recall caps

Fixes: #9284
Signed-off-by: John Spray <john.spray@redhat.com>

mds: add state for tracking RECALL progress

To be used later for generating health metrics
for clients which are failing to promptly service
CEPH_SESSION_RECALL_STATE messages.

Signed-off-by: John Spray <john.spray@redhat.com>

xlist: implement const_iterator

Signed-off-by: John Spray <john.spray@redhat.com>

client: fix trim_caps for inodes in root

Previously client would fail to release caps for files
in the root directory in response to CEPH_SESSION_RECALL_STATE
messages.

Signed-off-by: John Spray <john.spray@redhat.com>

client: failure injection for cap release

Used for simulating a buggy client that trips
the error detection in #9282 (warn clients
which aren't revoking caps)

Signed-off-by: John Spray <john.spray@redhat.com>

client: fix potentially invalid read in trim_caps

trim_dentry can potentially free an inode, so get/put
it around the block where we use the inode's dn_set.

Signed-off-by: John Spray <john.spray@redhat.com>

client: more precise cap trimming

Two fixes:
 * Client would unlink everything it could, instead of just
   meeting its goal, because caps.size() doesn't change until
   dentries are cleaned up later.  Take account of the trimmed
   count in the while() condition to fix that.
 * Don't count the root ino as trimmed, as although it has no
   dentries (of course), we will never give up the cap.

With this change, the client will now precisely achieve the number
of caps requested in CEPH_SESSION_RECALL_STATE messages.

Signed-off-by: John Spray <john.spray@redhat.com>

client: fix crash in trim_caps

In a75af4c2, procedure was added to invalidate root's dentries
if the trimming failed to free enough caps.  This would sometimes
crash because root->dir wasn't necessarily open.

Fix by only doing it if root dir is open, though I suspect this
may not be the end of it...

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #2485 from Abioy/master

bugfix: wrong socket address in log msg of Pipe.cc

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

bugfix: wrong socket address in log msg of Pipe.cc

paddr was not yet set up for the socket address

Signed-off-by: Yongyue Sun abioy.sun@gmail.com

Merge pull request #2442 from dachary/wip-6754-jerasure-parameters

erasure-code: fix BlaumRoth sanity check on w

Reviewed-by: Andreas Peters <andreas.joachim.peters@cern.ch>

Merge pull request #2488 from cernceph/docfix

doc: osd_backfill_scan_(min|max) are object counts

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

doc: osd_backfill_scan_(min|max) are object counts

osd_backfill_scan_min and osd_backfill_scan_max set the number of
items grabbed during a single backfill scan, not an interval in
seconds. Correct the doc.

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

rbd: ObjectCacher reads can hang when reading sparse files

The pending read list was not properly flushed when empty objects
were read from a space file.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

init-radosgw.sysv: Support systemd for starting the gateway

When using RHEL7 the radosgw daemon needs to start under systemd.

Check for systemd running on PID 1. If it is then start
the daemon using: systemd-run -r <cmd>. pidof returns null
as it is executed too quickly, adding one second of sleep and
script reports startup correctly.

Signed-off-by: JuanJose 'JJ' Galvez <jgalvez@redhat.com>

Merge pull request #2484 from sjahl/master

doc: Added bucket management commands to ops/crush-map

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

doc: Added bucket management commands to ops/crush-map

Describes the CLI for adding and removing buckets, in addition to the
'moving' instructions which were already present.

Signed-off-by: Stephen Jahl <stephenjahl@gmail.com>

Merge remote-tracking branch 'gh/giant'

Merge pull request #2481 from sjahl/master

doc: fixes a formatting error on ops/crush-map

doc: fixes a formatting error on ops/crush-map

Signed-off-by: Stephen Jahl <stephenjahl@gmail.com>

Merge pull request #2467 from majianpeng/fix3

buffer: In rebuild_page_aligned for the last ptr is page aligned, no need call rebuild().

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2478 from ceph/wip-9445

global: fix hang when segv happens inside logging code

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2477 from ceph/wip-client-msg-leak

client: fix a message leak

mds: update segment references during journal rewrite

... to avoid leaving log events that reference log
segments by offsets which no longer exist.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 386f2d7c829422695a1b1f41bd3f17ca3eef1f61)
Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #2469 from ceph/wip-9427-rewrite

mds: update segment references during journal rewrite

Reviewed-by: Greg Farnum <greg@inktank.com>

log: add simple test to verify an internal SEGV doesn't hang

Test that the segv injection works.

Test that a segv while logging something doesn't hang when the signal
handlers are installed.  Note that this fails/hangs without the previous
fix.

Signed-off-by: Sage Weil <sage@redhat.com>

client: fix a message leak

Signed-off-by: John Spray <john.spray@redhat.com>

global/signal_handler: do not log if SEGV originated inside log code

Signed-off-by: Sage Weil <sage@redhat.com>

log: add Log::is_inside_log_lock()

Signed-off-by: Sage Weil <sage@redhat.com>

mds: update segment references during journal rewrite

... to avoid leaving log events that reference log
segments by offsets which no longer exist.

Signed-off-by: John Spray <john.spray@redhat.com>

rgw: push hash calculater deeper

This might have been the culprit for #9307. Before we were calculating
the hash after the call to processor->handle_data(), however, that
method might have spliced the bufferlist, so we can't be sure that the
pointer that we were holding originally is still invalid. Instead, push
the hash calculation down. Added a new explicit complete_hash() call to
the processor, since when we're at complete() it's too late (we need to
have the hash at that point already).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

osd_types: fix pg_log_t tail dump

Fixes: #9453
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2471 from ceph/wip-9446

mon: fix MDS health detail output

Reviewed-by: Sage Weil <sage@redhat.com>

erasure-code: fix erasure_code_benchmark goop (decode)

Using a stringstream that is only displayed on error when calling the
erasure code factory, instead of cerr. The user expects the output to be
clean when there is no error. That was done for the encode function but
not the decode function.

http://tracker.ceph.com/issues/9429 Fixes: #9429

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

mon: fix MDS health detail output

I fat fingered a couple of things here.

Signed-off-by: John Spray <john.spray@redhat.com>

buffer: Add a test for bufferlist::rebuild_page_aligned

Make the last prt of bufferlist which is page-aligned don't change anything
after rebuild_page_aligned.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>

buffer: In rebuild_page_aligned for the last ptr is page aligned, no need call rebuild().

This only happen for the last ptr. Because rebuild() don't change the len
of ptr, so if last ptr isn't page-size aligned but is page aligned, the
rebuild() don't change anything.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>

Merge pull request #2468 from dachary/wip-always-create-pidfile

daemons: write pid file even when told not to daemonize

Reviewed-by: Loic Dachary <loic-201408@dachary.org>
Reviewed-by: Sage Weil <sage@redhat.com>

daemons: write pid file even when told not to daemonize

systemd wants to run daemons in foreground, but daemons wouldn't write
out the pid file with -f.  Fixed.

Signed-off-by: Alexandre Oliva <oliva@gnu.org>

Merge pull request #2464 from dachary/wip-9429-bench

erasure-code: fix erasure_code_benchmark goop

Reviewed-by: Janne Grunau <j@jannau.net>

Merge pull request #2416 from xiaoxichen/make_crush_private

Change CrushWrapper::crush to private

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2450 from dachary/wip-9413-erasure-code-version-check

erasure-code: mon, osd etc. depend on the plugins

ReplicatedPG: cancel cb on blacklisted watcher

Fixes: #8315
Backport: firefly
Signed-off-by: Samuel Just <sam.just@inktank.com>

erasure-code: fix erasure_code_benchmark goop

Using a stringstream that is only displayed on error when calling the
erasure code factory, instead of cerr. The user expects the output to be
clean when there is no error.

http://tracker.ceph.com/issues/9429 Fixes: #9429

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2409 from apeters1971/wip-ec-isa-table-cache-refac-master

EC-ISA: add intelligent table cache

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

doc: Fixed syntax error.

Signed-off-by: John Wilkins <jowilki@redhat.com>

doc: Updated authentication notes. Fixed syntax error.

Signed-off-by: John Wilkins <jowilki@redhat.com>

Merge pull request #2459 from ceph/wip-7934

test: Fix ceph_test_rados_watch_notify to delete the pools it creates

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2463 from ceph/wip-mds-beacon

mds: a couple fixes for the beacons

Reviewed-by: John Spray <john.spray@redhat.com>

mds: sleep in progress thread if laggy and waiting_for_nolaggy waiters

If we have nolaggy waiters but are laggy we want to sleep.  Otherwise,
we will just spin and spam the log ...

Signed-off-by: Sage Weil <sage@redhat.com>