ReplicatedPG: dump snap_trimq on pg query

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: do not queue the snap trimmer constantly

Previously, we continuously requeued the snap trimmer while in
TrimmingObjects.  This is not a good idea now that we try to
limit the number of snap trimming repops in flight and requeue
the snap trimmer directly as those repops complete.

Fixes: #9113
Backport: giant, dumpling, firefly
Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: clean out completed trimmed objects as we go

Also, explicitely maintain a max number of concurrently trimming
objects.

Fixes: 9113
Backport: dumpling, firefly, giant
Signed-off-by: Samuel Just <sam.just@inktank.com>

0.80

Merge pull request #1763 from ceph/wip-blacklist

Wip blacklist

Backport: firefly
Reviewed-by: Sage Weil <sage@inktank.com>

osd: check blacklisted clients in ReplicatedPG::do_op()

OSD checks if client is blacklisted only when receiving OSD request.
It's possible that OSD request's sender get blacklisted while OSD
request in in some waiting list.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

ceph-object-corpus: v0.80-rc1-35-g4812150

Signed-off-by: Sage Weil <sage@inktank.com>

mon/PGMonitor: set tid on no-op PGStatsAck

The OSD needs to know the tid.  Both generally, and specifically because
the flush_pg_stats may be blocking on it.

Fixes: #8280
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
(cherry picked from commit 5a6ae2a978dcaf96ef89de3aaa74fe951a64def6)

mon/OSDMonitor: share latest map with osd on dup boot message

If we get a dup boot message, share the newer maps with the osd so that
they know they are living in the past.

Fixes: #8279
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
(cherry picked from commit 2e6b24868da0b203c2d70ac91071166d95d1d851)

Merge pull request #1751 from ceph/wip-mds-shutdown

mds: remove mdsdir in the final step of shutdown MDS

Reviewed-by: Sage Weil <sage@inktank.com>

mds: remove mdsdir in the final step of shutdown MDS

Otherwise we may get bad subtree map if we restart the MDS before
the shutdown process finishes.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

ReplicatedPG: block scrub on blocked object contexts

Fixes: #8011
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit e66f2e36c06ca00c1147f922d3513f56b122a5c0)

rados.h,ReplicatedPG: add CEPH_OSD_FLAG_ENFORCE_SNAPC and use on flush

We need to ensure that even with pool snaps, we use the snapc provided in order
to ensure that the clones are written back correctly.

Fixes: #7941
Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 499adb1db1cd225c91acce31b5f48fad5145043b)

ECBackend::continue_recovery_op: handle a source shard going down

get_min_avail_to_read_shards might return an error if there are
no longer enough sources to reconstruct the missing shards.
This is possible if osds went down while we were writing the
previous chunk -- we already notice in check_recovery_sources
if a source goes down during a read.

Fixes: #8161
Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 1885792c517670086332a8bab237c58558ee6dda)

ReplicatedPG: we can get EAGAIN on missing clone flush

Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 060105c313c5b4a777c55f17115eeb95ebb17117)

ReplicatedPG: do not preserve op context during flush

Any information stashed in the OpContext may be obsolete by the time we
actually mark the object clean.  Instead, let the start_flush caller
clean up its OpContext and in try_flush_mark_clean we'll create a new
one.  The primary reason to keep the OpContext would have been locking,
but we can set the obc as blocking without holding an OpContext, and
that would allow trimming to happen in the mean time (which is good
since trim_object does not respect rw locks since it doesn't change user
visible state).  In try_flush_mark_clean, we requeue the fop->op along
with (but ahead of) the fop->dup_ops.

Fixes: #8068
Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit d83b8f58513e86c68aa6c25d4f909d9a3c4a103e)

rgw: fix url escaping

Fixes: #8202
This fixes the radosgw side of issue #8202. Needed to cast value
to unsigned char, otherwise it'd get padded.

Backport: dumpling

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit bcf92c496aba0dfde432290fc2df5620a2767313)

doc/release-notes: v0.67.8 notes

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1729 from ceph/wip-7966

readlink result in resapwn

Reviewed-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: terminate readlink result in resapwn

readlink(2) does not null terminate the buffer; we need to do that.

Fixes: #7966
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1727 from ceph/wip-8193

ceph_test_rados_api_tier: increase HitSetTrim timeouts

Merge pull request #1725 from FlorentCoppint/master

Skipping '_netdev' Debian fstab option

Reviewed-by: Sage Weil <sage@inktank.com>

ceph_test_rados_api_tier: increase HitSetTrim timeouts

...so that they pass when they get unlucky with thrashing.

This will vastly decrease the probability of failure, but failure will
always be possible when a timeout is in place.

Fixes: #8193
Signed-off-by: Sage Weil <sage@inktank.com>

Skipping '_netdev' Debian fstab option

Signed-off-by: Florent Bautista <florent@coppint.com>

Merge pull request #1717 from dachary/wip-auid

mon: add ceph osd pool set <pool> auid

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #1724 from ceph/wip-uselocalgithubforqemu-wusui

Use new git mirror for qemu-iotests

Use new git mirror for qemu-iotests

Fixes: 8191
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Merge remote-tracking branch 'gh/firefly'

Merge pull request #1720 from jdurgin/wip-list-children-test

test_rbd.py: ignore children in cache pools

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1719 from ceph/wip-8168

Wip 8168

Reviewed-by: Sage Weil <sage@inktank.com>

ReplicatedPG::do_osd_ops: consider head whiteout in list-snaps

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::do_op: don't return ENOENT for whiteout on snapdir read

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1718 from ceph/wip-7882-wusui

Support latest qemu iotest code

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

test_rbd.py: ignore children in cache pools

This is necessary until http://tracker.ceph.com/issues/8187 is fixed.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

mon: add ceph osd pool set <pool> auid

When a pool is created with ceph osd pool create, the auid is not
inferred from the session auid and is set to zero. Add the

  ceph osd pool set <pool> auid <int>

command to set it after it is created, and the matching get:

  ceph osd pool get <pool> auid

Signed-off-by: Loic Dachary <loic@dachary.org>

Support latest qemu iotest code

Modified qemu-iotests workunit script to check for versions
that use the latest qemu (currently only Trusty).  Limit the
tests to those that are applicable to rbd.

Fixes: 7882
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Merge pull request #1714 from ceph/wip-fs-client

two small fixes for client

Reviewed-by: Sage Weil <sage@inktank.com>

client: check cap ID when handling cap export message

handle following sequence of events:
- mds0 exports an inode to mds1. client receives the cap import
  message from mds1. caps from mds0 are removed while handling
  the cap import message.
- mds1 exports an inode to mds0. client receives the cap export
  message from mds1. handle_cap_export() adds placeholder caps
  for mds0
- client receives the first cap export message (for exporting
  inode from mds0 to mds1)

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

client: avoid releasing caps that are being used

To avoid releasing caps that are being used, encode_inode_release()
should send implemented caps to MDS.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

Merge pull request #1713 from ceph/wip-7439

Wip 7439

Reviewed-by: Sage Weil <sage@inktank.com>

ReplicatedPG: handle ec pools in mark_all_unfound_lost

Signed-off-by: Samuel Just <sam.just@inktank.com>

0.80-rc1

ReplicatedPG: enable mark_unfound_lost delete for ec pools

revert is tricky to implement at this time for ec pools, so
we'll instead just implement delete for ec pools.

Fixes: #7439
Signed-off-by: Samuel Just <sam.just@inktank.com>

qa/workunits/rbd/copy.sh: skip some tests when tiering is enabled

The rados ls bit doesn't work.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

qa/workunits/rbd/copy.sh: fix test

I broke this in commit 9d64ac66082bd108ec3c2a74e2e77475b5564eae.

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1691 from ceph/wip-8139

osd_types: pg_t: allow is_split to handle checks for splits prior to the most recent

Reviewed-by: Samuel Just <sam.just@inktank.com>

ECBackend: use std::swap for boost::optional

Reviewed-by: Sage Weil <sage@inktank.com>
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1710 from ceph/wip-coverity

a couple coverity fixes

Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1711 from ceph/wip-coverity-respawn

mds: make strncpy in ::respawn safer

Reviewed-by: Sage Weil <sage@inktank.com>

mds: make strncpy in ::respawn safer

Previous code assumed null terminated argv[0]
was not longer than PATH_MAX and the resulting
strncpy was not strictly safe.

Modify the bounds to ensure that copy will not
result in an unterminated string if argv[0]
is oversized.

Signed-off-by: John Spray <john.spray@inktank.com>

osd/osd_types: RWState: initialize snaptrimmer_write_marker

** CID 1204295:  Uninitialized scalar field  (UNINIT_CTOR)
/osd/osd_types.h: 2716 in ObjectContext::RWState::RWState()()

Signed-off-by: Sage Weil <sage@inktank.com>

osdc/Objecter: drop unused field

This as missed by 860d72770cdf092c027d50f4ee03bed76c975599.

** CID 1204296:  Uninitialized scalar field  (UNINIT_CTOR)
/osdc/Objecter.h: 1165 in Objecter::Op::Op(const object_t &, const
object_locator_t &, std::vector<OSDOp, std::allocator<OSDOp>> &, int, Context *,
Context *, unsigned long *)()
/osdc/Objecter.h: 1165 in Objecter::Op::Op(const object_t &, const
object_locator_t &, std::vector<OSDOp, std::allocator<OSDOp>> &, int, Context *,
Context *, unsigned long *)()

Signed-off-by: Sage Weil <sage@inktank.com>

doc/release-notes: a bit of prose about firefly

Signed-off-by: Sage Weil <sage@inktank.com>

osd/osd_types: pg_interval_t: include primaries in operator<<

Also make up vs acting explicit.

Signed-off-by: Sage Weil <sage@inktank.com>

osd/osd_types: pg_interval_t: include up_primary in pg_interval_t

Nothing uses this, but it triggers a new interval, which makes it confusing
when it is not recording in the interval itself.  Let's add it now.

Signed-off-by: Sage Weil <sage@inktank.com>

osd/osd_types: pg_interval_t: dump primary

Signed-off-by: Sage Weil <sage@inktank.com>

osd: change in up set primary constitutes a peering interval change

In several places, a change in the up_primary triggers a new peering
interval, but the palces that actually generate the new past intervals,
including check_new_interval(), did not enforce that.  This becomes
somewhat obvious when you see that those callers are ignoring the
up_primary output argument for pg_to_up_acting_osds().

Fix this by adding arguments to check_new_interval and fixing the callers
to pass them in properly.  Add a unit test case to verify this.

Note that the past interval struct itself does not record who the
up_primary was; possibly it should.

Fixes: #8139
Signed-off-by: Sage Weil <sage@inktank.com>

osd: use parent pgid (as appropriate) in generate_past_intervals()

Feed in the ancestor pg_t (if any) when we are looking at intervals for
previous maps that may have preceded a recent split.

Fixes: #8139
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1651 from enovance/wip-brag

Few bug fixes in ceph-brag

Reviewed-by: Sage Weil <sage@inktank.com>

doc/release-notes: v0.80

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1707 from ceph/wip-rbd-test

rbd: fix tests for cache pools

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

qa/workunit/rbd/import_export.sh: skip list-objects tests with tiering

Listing objects isn't reliable with cache pools; skip that part of the
test if we see that rbd has tiering enabled.

Signed-off-by: Sage Weil <sage@inktank.com>

qa/workunit/rbd/copy.sh: do not delete/recreate rbd pool

Among other things, it breaks when tiering is enabled.

Signed-off-by: Sage Weil <sage@inktank.com>

doc: Fixed syntax to include 'pool'.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

PG::PriorSet: consider lost osds in up_now for pcontdec

Otherwise, the pg will remain down even as osds are marked lost.

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1703 from ceph/wip-7942

Wip 7942

Reviewed-by: Sage Weil <sage@inktank.com>

ReplicatedPG::do_op: check for blocked snapset obj

Otherwise, we might use an invalid snapset in find_object_context.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: in trim, grab w locks on obc and snapset_obc

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: if we get ENOENT on clone, remove clone from snapset

Fixes: #7916
Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: do not create whiteout clones

First, make_writeable treats whiteout heads like snapdir for
cloning purposes.  Second, to ensure that we send the correct
deletes on flush to the backing pool, we instead use oi.snaps
on any clone we are flushing to infer the snaps during which
head did not exist and send a delete as appropriate prior to
the copy_from.

Normally, we'd have a problem if the delete and the copy_from
completed, but an interval change intervened before the dirty
flag was cleared since we'd end up re-deleting the object.
To avoid that, we use the CEPH_OSD_FLAG_ORDERSNAP flag.

Additionally, we will use the correct snap_seq on the delete
or flush as appropriate to ensure that the previous clone
gets created with the same clone id as in the cache pool.

Fixes: #7942
Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG,rados: add CEPH_OSD_[COPY_FROM]_MAP_SNAP_TO_CLONE

When promoting a clone, we want to use the provided snapid to specify
specify the clone id directly.

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1705 from ceph/wip-8124

Wip 8124

Reviewed-by: Sage Weil <sage@inktank.com>

qa/workunits/cephtool/test.sh: make set pg_num test non-racy

Loop while the pool is creating.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

ReplicatedPG: do not use shard for hit_set object names

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::agent_load_hit_sets: take ondisk_read_lock

Otherwise, the hit_set might be not yet written due to a recently
completed recovery.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG,PGLog: update hit_set during peering

Fixes: #8124
Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: propogate hit_set history with repop

We don't actually send the whole info on each repop, just the log
entries, updated stats, and a few other bits.  For hit_set ops, we need
to also communicate the new hit_set history status atomically with the
log entries and the transaction.  Thus, we add a channel for an optional
pg_hit_set_history_t field in PGBackend::submit_transaction interface
and associated messages and implementations to update the hit_set info
field along with the log entries.

This also means that hit_set_(persist|trim) update an
updated_hit_set_history field on the OpContext instead of directly
modifying the info field.

Fixes: #8124
Signed-off-by: Samuel Just <sam.just@inktank.com>

encoding: use unqualified name for encode/decode in boost::optional encoding

Signed-off-by: Samuel Just <sam.just@inktank.com>

ECMsgTypes::ECSubWrite: fix at_version indentation

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd: track the number of hit_set archive objects in a pg

Also, use this value in agent_choose_mode instead of the max
number.

Related: #8124
Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::hit_set_persist: clean up degraded check

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::mark_all_unfound_lost: delete local copy if necessary

There might be a local copy for an EC pool in the DELETE case.  The replica
copies should be already handled by merge_log.

Fixes: #7439
Signed-off-by: Samuel Just <sam.just@inktank.com>

buffer: adjust #include order

The pthread.h include is somehow clobbering things, although it is not
clear how.  :(

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1696 from ceph/wip-8097

buffer: use Mutex instead of Spinlock for raw crcs

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1697 from ceph/wip-num_objects_omap

osd_types::object_stat_sum_t: fix add/sub for num_objects_omap

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1695 from ceph/wip-8153

Revert "ReplicatedPG::get_snapset_context: assert snap obj is not missin...

Reviewed-by: Sage Weil <sage@inktank.com>

Revert "ReplicatedPG::get_snapset_context: assert snap obj is not missing"

This breaks mark_lost_unfound_revert.

This reverts commit 0d2177a18071ad9c9581826a43751c36bab5b2db.

Merge pull request #1693 from ceph/wip-7997

mon: fix get_version race (more)

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

test: handle the create-pg delay when testing cache split syntax

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1692 from ceph/wip-7784

mon: OSDMonitor: HEALTH_WARN on 'mon osd down out interval == 0'

Reviewed-by: Sage Weil <sage@inktank.com>

mon: OSDMonitor: HEALTH_WARN on 'mon osd down out interval == 0'

A 'status' or 'health' request will return a HEALTH_WARN whenever the
monitor handling the request has the option set to zero.

Fixes: 7784
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: wait for PaxosService readable in handle_get_version

We were waiting for the election to finish, but we need to *also* wait for
paxos to recover.  Being a peon or leader is not sufficient and we may
return a map that is still old.

Fixes: #7997
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1676 from ceph/wip-8092

Wip 8092

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1678 from ceph/wip-8108

osd: OSDMap: have osdmap json dump print valid boolean instead of string

Reviewed-by: Sage Weil <sage@inktank.com>

osd_types: pg_t: add get_ancestor() method

Give us the ancestor for when the pool had a past value for pg_num.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1683 from ceph/wip-mds-op-prio

mds: dynamically adjust priority of committing dirfrags

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #1689 from ceph/wip-8091

Wip 8091

Reviewed-by: Sage Weil <sage@inktank.com>

ReplicatedPG::recover_replicas: do not recover clones while snap obj is missing

Otherwise, we cannot safely read the snapset for the clone.

Fixes: #8091
Signed-off-by: Samuel Just <sam.just@inktank.com>

osd_types::object_stat_sum_t: fix add/sub for num_objects_omap

Introduced in a130a4452e4fb159dc62fb417077d98dc9ebd621
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1688 from ceph/wip-8048

osd/ReplicatedPG: check clones for degraded

Reviewed-by: Samuel Just <sam.just@inktank.com>