librbd: check for presence of journal before attempting to remove

Fixes: #13912
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: helper method to determine if journal exists

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6625 from dillaman/wip-12698

librbd: integrate journaling for maintenance operations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6687 from dillaman/wip-journal-replay-fixes

journal: support replaying beyond skipped splay objects

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6685 from dachary/wip-erasure-code-benchmark

qa: erasure-code benchmark plugin selection

Reviewed-by: Andreas Peters <andreas.joachim.peters@cern.ch>

Merge pull request #6292 from dx9/wip-12406-res_nquery

rgw/rgw_resolve: fallback to res_query when res_nquery not implemented

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6236 from guangyy/err-msg

osd: use pg id (without shard) when referring the PG

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6610 from ktdreyer/wip-build-doc-lxml

admin/build-doc: add lxml dependencies on debian

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6323 from dingshang/wip-cephfs-dingshang

pybind/cephfs: add symlink and its unit test

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge tag 'v10.0.0'

v10.0.0

Merge pull request #6684 from jcsp/wip-fix-scrub

mds: fix scrub_path

journal: support replay passed skipped splay objects

It's possible for a splay object within a set to be skipped
if the set is closed due to a full object within the set.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: verify that journal player can handle skipped journal objects

It's possible for a journal object to not exist if another journal object
within the same object set filled up before records were written.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6605 from yuyuyu101/wip-13797

ceph_test_msgr: Use send_message instead of keepalive to wakeup connection

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6495 from objoo/master

Mailmap updates for infernalis.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: Jenkins affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Burkhard Linke affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Chen Dihao affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Wei Qian affiliation
Signed-off-by: Yann Dupont <yann@objoo.org>

qa: erasure-code-benchmark technique and plugin selection

Update the PLUGINS variable that was no longer used. Add the TECHNIQUES
variable to control which techniques are compared.

Signed-off-by: Loic Dachary <loic@dachary.org>

qa: erasure-code has --erasure-code-dir

It is used instead of the obsoleted --parameter directory= to specify
the location of the erasure code directory plugins.

Signed-off-by: Loic Dachary <loic@dachary.org>

mds: fix scrub_path

This was tripping up over calling
validate_disk_state with no ScrubHeader.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #6679 from suckowbiz/patch-1

Fixed typos

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/release-notes: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: final v10.0.0 notes

Signed-off-by: Sage Weil <sage@redhat.com>

doc: fix message typos in systemd

Signed-off-by: Tobias Suckow <tobias@suckow.biz>

Merge branch 'master' of github.com:ceph/ceph

Merge pull request #6666 from dachary/wip-release-notes

release-notes: draft v10.0.0 release notes

Merge branch 'wip-bigbang'

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/mon/osd-crush.sh: escape ceph tell mon.*

Signed-off-by: Sage Weil <sage@redhat.com>

osd: make some of the pg_temp methods/fields private

Reported-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: call notify completion only once

If we race with a reconnect we could get a second notify message
before the notify linger op is torn down.  Ensure we only ever
call the notify completion once to prevent a segfault.

Fixes: #13805
Signed-off-by: Sage Weil <sage@redhat.com>

mon: change mon_osd_min_down_reporters from 1 -> 2

This makes more sense to me.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: simplify failure reporters vs reports logic

Since each OSD only sends a failure report for a given peer once,
we don't need to count reports vs reporters separately.  (This was
probably a bad idea anyway.)  Remove this logic and the associated
config option.

Reported-by: Greg Farnum <gfarnum@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>

osd: simplify pg creation

We used to have a complicated pg creation process in which we
would query any previous mappings for the pg before we created the
new 'empty' pg locally.  The tracking of the prior mappings was
very simple (and broken), but it didn't really matter because the
mon would resend pg create messages periodically.  Now it doesn't,
so that broke.

However, none of this is necessary: the PG peering process does
all of the same things.  Namely, it

- enumerates past intervals
- determines which ones may have been rw
- queries OSDs from each one to gather any potential changes

This is a more robust version of what the creation code was (or
should have been doing).  So, let's rip it all out and let
peering handle it.  As long as the newly instantiated PG sets
last_epoch_started and _clean to the created epoch we will probe
and consider all of these prior mappings and find any previous
instance of the PG (if one existed).

Yay for removing unnecessary code!

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: make _sub_got behave if we "got" old stuff

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: fix oldest_map in send_incremental

This should be the oldest map on the sender (like every other
place that generates an MOSDMap message).

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: avoid useless pg gets when pool is deleted

If the .0 pg no longer exists, we know the entire pool was
deleted, and can avoid querying every other pg.  (This is a good
thing because leveldb and rocksdb can be very slow to query
missing keys.)

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: revamp how pg creates are tracked

Previously we were calculating and managing in-core state that
wasn't committed as part of the pg_map, leading to all sorts of
ugliness that didn't really work.  Instead,

 * set mapping in all creating pgs in the committed pg_map
 * make all pg create message sending be based on committed state
 * update mappings for creating pgs every time we consume a new
   osdmap, so that we have a reliable/stable epoch to attach to
   it.

In particular, having that stable epoch means we have a reference
we can put in the pg create message that will also be used for
the subscription version.  That way OSDs get consistent creates
from any mon.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: only send pg create messages to up osds

If the OSD is down it will ignore the message.  If it gets marked up, we
will eventually consume that map and call check_subs().

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: only churn mapping_epoch if the primary changes

This results is fewer resent pg create messages.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: a bunch of cosmetic cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: drop old creating_pgs_by_osd

Obsoleted by creating_pgs_by_osd_epoch.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: reduce mon_subscribe messages

1. MonClient remembers our subscriptions; only indicate we want
osd_pg_creates once, in init.

2. We don't need to re-request the latest osdmap each time we
reconnect.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: only send new subscriptions

Instead of resending all subscriptions, only send the new ones.  This
avoids races like

 - ask for 4+
 - mon sends maps 4-50
 - ask for 4+ and something else
 - mon has to resend same maps and the other thing

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: send pg creates via persistent subscriptions, not spam

Generate and send pg create messages only for those OSDs who have
subscribed on this monitor.  This is N time more efficient (where there
are N monitors) than the previous method.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: only map and send pg creates post paxos update

These other call sites are no longer needed.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: remove map_pg_creates, send_pg_creates commands

These shouldn't be triggered manually.

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDPGCreate: make it more readable

1- include the epoch
2- drop the 'pg'
3- hide the timestamp

Signed-off-by: Sage Weil <sage@redhat.com>

osd: subscribe to all pg creates, not just once on start

We want to know about all future pg creations, not just those pending
when we start.  (This only helps once the mon knows how to do this...)

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: track creating_pgs_by_osd_epoch

Track pg creations, grouped by the first epoch they mapped to a particular
OSD.  This will be necessary to send messages only for new creations.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: assert our pg counts don't go negative

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: do not prime pg_temp for creating pgs

It will be less work for the old primary to ignore the create message
and the new one to query it and find nothing that for the slightly more
complicated peering and removal process to happen.  Also, this reduces
bloat in the OSDMap a bit.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: note mapping_epoch for creating pgs

Signed-off-by: Sage Weil <sage@redhat.com>

mon: let peon mons send the osdmap replies

Currently the leader mon often replies to OSDs by sending a set of
incremental OSDmaps (e.g., in response to an osd boot or failure).

Instead, send a small message to the proxying peon mon (if any)
with the epoch to start from and let *them* generate a suitable
reply.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/simple/Pipe: show keepalives at level 2

Signed-off-by: Sage Weil <sage@redhat.com>

mon: set mon_subscribe_interval to a day

This is only needed for legacy clients to avoid confusing them--
we don't actually need the renewals at all.  Make them infrequent
to reduce mon load.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: only ack subscriptions (and renew) if client or mon is old

Old client expect an ack so they can schedule renewal; send it for
them only.

Old mons expect renewals.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: remove old subscribe renewal-based timeouts

This is no longer needed/used.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: small cleanup in _ms_dispatch

Signed-off-by: Sage Weil <sage@redhat.com>

mon: new session_timeout mechanism that is not subscribe-based

Simplify the session liveness detection:

 - renew on any message
 - renew on keepalive[2] messages (lightweight ping in msgr)

Signed-off-by: Sage Weil <sage@redhat.com>

msg: make last_keepalive[_ack] lock safe

Signed-off-by: Sage Weil <sage@redhat.com>

msg: track stamp of last keepalive[2] received

Signed-off-by: Sage Weil <sage@redhat.com>

common: mirror leveldb default tuning w/ rocksdb

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: don't send log if we're reconnecting

Signed-off-by: Sage Weil <sage@redhat.com>

mon: disabled rocksdb compression when used as the backend

This significantly reduced CPU utilization on the bigbang scale
testing cluster at CERN.  Note that it is already disabled for
leveldb by default (in ceph_mon.cc).

Signed-off-by: Sage Weil <sage@redhat.com>

osd: cap adjusted max mon report interval at 2/3 of timeout

This ensures that we don't throttle back mon reports so much that
the mon times out out due to no pg stat reports.  Since there is
little value is having a lower max anyway, just set this at an
upper bound (relative to the mon's timeout value).

Signed-off-by: Sage Weil <sage@redhat.com>

osd: protect mon reporting with mon_report_lock

We need an exclusive lock over paths that update state related to
mon reports, lest they step on fields like up_thru_*, *stats_ack*,
last_mon_report, and so on.  Everybody still needs a read lock
on map_lock too to get a stable OSDMap epoch.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fix reconnect behavior from booting state

We don't need to restart the boot process unless we are in preboot;
if we are in booting state we just need to resend the boot
message.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move the monitor report to OSD::tick_without_osd_lock

Fixes: #12722
Reviewed-by: Guang Yang <yguang@yahoo-inc.com>

osd: _got_mon_epochs - refactor the lock scope to avoid a race (which fail make check)

Reviewed-by: Guang Yang <yguang@yahoo-inc.com>

osd: don't send dup subscribes so much

The subscribe MonClient service is stateful--we don't need to
force a new subscribe send unless sub_want() says we need to.

Keep forcing it for instances where we request an *old* map.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: introduce explicit preboot stage

We want to separate the stage where we do a bunch of work
prior to booting (but intend to eventually boot), like when we
get maps and wait to be healthy, from the point after we've sent
the boot message while we are just waiting for a response (so that
we can avoid resending that boot message needlessly).

- start at PREBOOT in start_boot()
- transition to BOOTING in _send_boot()
- only call _preboot() while in PREBOOT state

Signed-off-by: Sage Weil <sage@redhat.com>

osd: skip osdmap version query if we can

If we get OSDmaps from the mon we *also* learn the oldest/newest
map epochs; no need to query again.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: make [_]maybe_boot lockless variant

Signed-off-by: Sage Weil <sage@redhat.com>

osd: only send boot if booting on getversion completion

Signed-off-by: Sage Weil <sage@redhat.com>

osd: do not resend pg_temp requests

Send each pg_temp request once (per mon session); no need to
resend everything that is pending every time.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: do not send dup failure reports

If a failure report is already pending, we do not need to resend
it.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: resend pending failure reports with a new mon session

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fix send_failures() locking

It is unsafe to check failure_queue.empty() without the lock.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: backoff the max reporting interval, too

Signed-off-by: Sage Weil <sage@redhat.com>

osd: no need for regular send_pg_temps

This is done by process_peering_events.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: just send alive when it is queue

No need to futz with last_mon_report or resend it again later.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fix pg stat reporting

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6565 from chenji-kael/patch-1

Update .organizationmap

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6673 from dzafman/wip-rand-scrub-fix

Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd: Only add random deep scrubs when NOT user initiated scrub

Signed-off-by: David Zafman <dzafman@redhat.com>

Revert "test: osd-scrub-snaps.sh: Randomized deep-scrubs can now happen during a scrub"

This reverts commit 0fe26c25c5fefd12628fcdba67be047f640b4afc.

Merge pull request #6219 from joejulian/master_fix_13438_parted_must_contain_dev

Compare parted output with the dereferenced path

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6671 from dzafman/wip-jenkins-fix

Fix Jenkins make check errors due to deep-scrub randomization

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test: osd-scrub-snaps.sh: Randomized deep-scrubs can now happen during a scrub

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #6659 from cholcombe973/master

Doc: Typo in the apt-get command

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Typo in the apt-get command.
Signed-off-by: Chris Holcombe <xfactor973@gmail.com>

Merge remote-tracking branch 'gh/jewel'

Merge pull request #6553 from ukernel/wip-13777

mds: properly set STATE_STRAY/STATE_ORPHAN for stray dentry/inode

Reviewed-by: John Spray <john.spray@redhat.com>

scripts: ceph-release-notes for development versions

Relax the requirements for titles and issues when --strict is not set.
Collect and display the merge messages when they are not the same as the
PR title.

Signed-off-by: Loic Dachary <loic@dachary.org>

release-notes: draft v10.0.0 release notes

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #6603 from ukernel/wip-13782

client: avoid creating orphan object in Client::check_pool_perm()

Reviewed-by: John Spray <john.spray@redhat.com>

librbd: commit journal op events immediately

Op events are split into a start and finish record, so they can
be immediately marked as committed within the journal.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge branch 'osd-op-create' of git://github.com/majianpeng/ceph

Conflicts:
src/osd/ReplicatedPG.cc

Reviewed-by: Sage Weil <sage@redhat.com>