librbd: Fix rebase with new io flow

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: fix lttng tracing argument mismatch

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: Add set_event_notify to AioImageRequestWQ

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: Remove unneeded set_event_notify

Since xlist clear method invoked below is enough to disassociate the xlist
item from the xlist, so the remove_myself() call in the destructor is safe
to invoke since it's already been removed from the list.

Signed-off-by: Haomai Wang <haomai@xsky.com>

Makefile: Add noinst headr files

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: Make rbd header file uses independent enum definition

Signed-off-by: Haomai Wang <haomai@xsky.com>

Librbd: Make AioCompletion complete doesn't unlock if callback

Signed-off-by: Haomai Wang <haomai@xsky.com>

Librbd: fix return code of EventSocket init and notify

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: Add ictx check to avoid AIO_TYPE_NONE completion

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: normalize notify return code

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: Fix incorrect api declaration

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: check event_notify to avoid extra logic

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

EventSocket: Add new event type pipe support

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

test: Add tests for getting arg of completion

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: Add interface to let user can get private data from comp

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

tests: Add tests for user io event notify

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: Add event notify interfaces

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

EventSocket: Add EventSocket structure used for event notification

EventSocket will wrap different user event notification method like linux
eventfd, solaris port. Caller can user this to replace signal

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #6533 from ghost/wip-fix-trivial-bug

osd: fix trivial scrub bug

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6612 from H3C/wip-yrf-destroy_collection

osd: fix FileStore::_destroy_collection error return code

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #6660 from chengyli/master

mon: fix ceph df pool available calculation for 0-weighted OSDs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6473 from H3C/wip-osd-bugfix1

auth: fail if rotating key is missing (do not spam log)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6278 from XinzeChi/wip-failinfo-mon

osd: cancel failure reports if we fail to rebind network

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6675 from rohanmars/wip-aix-librados-port

aix gcc librados port

Reviewed-by: Sage Weil <sage@redhat.com>

aix shared library build

Signed-off-by: Rohan Mars <code@rohanmars.com>

Merge branch 'wip-13800' of git://github.com/ukernel/ceph

client: fix deadlock related to async pagecache invalidation

Conflicts:
src/client/Client.cc
Fixed a conflict with the earlier page cache invalidate
changes in 73beb7f9378182cc3901fe86c4f1f5d5d98169a6.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #6454 from H3C/wip-mds

mds: repair the command option "--hot-standby"

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #6269 from jcsp/wip-client-mark-down

client: close mds sessions in shutdown()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #6380 from ukernel/wip-client-keep-cache

client: don't invalidate page cache when inode is no longer used

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #6253 from jcsp/wip-client-availability

client: a better check for MDS availability

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #6051 from clever215/master

rgw: add an inspection to the field of type when assigning user caps

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add an inspection to the field of type when assigning user caps

Bug #13096

This modification adds the check to the field of type of a user's capality while previous versions set it any value. i.e. we limit the option of types to the 9 certain values, which are "users|buckets|metadata|usage|zone|bilog|mdlog|datalog|ops These 9 choosens are found in ceph documents and in source codes.

Signed-off-by: Kongming Wu <wu.kongming@h3c.com>

Merge pull request #6700 from dillaman/wip-librbd-32bit-support

librbd: simplify IO method signatures for 32bit environments

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: simplify IO method signatures for 32bit environments

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6625 from dillaman/wip-12698

librbd: integrate journaling for maintenance operations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6687 from dillaman/wip-journal-replay-fixes

journal: support replaying beyond skipped splay objects

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6685 from dachary/wip-erasure-code-benchmark

qa: erasure-code benchmark plugin selection

Reviewed-by: Andreas Peters <andreas.joachim.peters@cern.ch>

Merge pull request #6292 from dx9/wip-12406-res_nquery

rgw/rgw_resolve: fallback to res_query when res_nquery not implemented

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6236 from guangyy/err-msg

osd: use pg id (without shard) when referring the PG

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6610 from ktdreyer/wip-build-doc-lxml

admin/build-doc: add lxml dependencies on debian

Reviewed-by: Sage Weil <sage@redhat.com>

client: s/close_sessions/_close_sessions/

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #6323 from dingshang/wip-cephfs-dingshang

pybind/cephfs: add symlink and its unit test

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

client: close mds sessions in shutdown()

Usually this happens in unmount(), but when we
have instantiated Client without mounting (to
send MDS commands), we need to handle closing
any open sessions in shutdown as well.

This is the correct replacement for the mark_down()
call that was removed from handle_command_reply
in the last commit.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge tag 'v10.0.0'

v10.0.0

Merge pull request #6684 from jcsp/wip-fix-scrub

mds: fix scrub_path

journal: support replay passed skipped splay objects

It's possible for a splay object within a set to be skipped
if the set is closed due to a full object within the set.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: verify that journal player can handle skipped journal objects

It's possible for a journal object to not exist if another journal object
within the same object set filled up before records were written.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6605 from yuyuyu101/wip-13797

ceph_test_msgr: Use send_message instead of keepalive to wakeup connection

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6495 from objoo/master

Mailmap updates for infernalis.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: Jenkins affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Burkhard Linke affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Chen Dihao affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Wei Qian affiliation
Signed-off-by: Yann Dupont <yann@objoo.org>

qa: erasure-code-benchmark technique and plugin selection

Update the PLUGINS variable that was no longer used. Add the TECHNIQUES
variable to control which techniques are compared.

Signed-off-by: Loic Dachary <loic@dachary.org>

qa: erasure-code has --erasure-code-dir

It is used instead of the obsoleted --parameter directory= to specify
the location of the erasure code directory plugins.

Signed-off-by: Loic Dachary <loic@dachary.org>

add aix compile warning

Signed-off-by: Rohan Mars <code@rohanmars.com>

initialized backtrace variables

Signed-off-by: Rohan Mars <code@rohanmars.com>

mds: fix scrub_path

This was tripping up over calling
validate_disk_state with no ScrubHeader.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #6679 from suckowbiz/patch-1

Fixed typos

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/release-notes: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: final v10.0.0 notes

Signed-off-by: Sage Weil <sage@redhat.com>

mon: do not ignore a failure report cancellation form osd

do not ignore a failure report cancellation form osd even if it is
down.

Signed-off-by: Xinze Chi <xinze@xsky.com>

mon: fix osd failure info in mon

when the network adapter of node A run into error, osd in this node
would tell mon other osd's heartbeat is timeout also. So when rebind
fail after retry 3 times, the osd should cancel in-flight failure report
send to mon before.

Signed-off-by: Xinze Chi <xinze@xsky.com>

doc: fix message typos in systemd

Signed-off-by: Tobias Suckow <tobias@suckow.biz>

Merge branch 'master' of github.com:ceph/ceph

Merge pull request #6666 from dachary/wip-release-notes

release-notes: draft v10.0.0 release notes

Merge branch 'wip-bigbang'

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/mon/osd-crush.sh: escape ceph tell mon.*

Signed-off-by: Sage Weil <sage@redhat.com>

osd: make some of the pg_temp methods/fields private

Reported-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: call notify completion only once

If we race with a reconnect we could get a second notify message
before the notify linger op is torn down.  Ensure we only ever
call the notify completion once to prevent a segfault.

Fixes: #13805
Signed-off-by: Sage Weil <sage@redhat.com>

mon: change mon_osd_min_down_reporters from 1 -> 2

This makes more sense to me.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: simplify failure reporters vs reports logic

Since each OSD only sends a failure report for a given peer once,
we don't need to count reports vs reporters separately.  (This was
probably a bad idea anyway.)  Remove this logic and the associated
config option.

Reported-by: Greg Farnum <gfarnum@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>

osd: simplify pg creation

We used to have a complicated pg creation process in which we
would query any previous mappings for the pg before we created the
new 'empty' pg locally.  The tracking of the prior mappings was
very simple (and broken), but it didn't really matter because the
mon would resend pg create messages periodically.  Now it doesn't,
so that broke.

However, none of this is necessary: the PG peering process does
all of the same things.  Namely, it

- enumerates past intervals
- determines which ones may have been rw
- queries OSDs from each one to gather any potential changes

This is a more robust version of what the creation code was (or
should have been doing).  So, let's rip it all out and let
peering handle it.  As long as the newly instantiated PG sets
last_epoch_started and _clean to the created epoch we will probe
and consider all of these prior mappings and find any previous
instance of the PG (if one existed).

Yay for removing unnecessary code!

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: make _sub_got behave if we "got" old stuff

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: fix oldest_map in send_incremental

This should be the oldest map on the sender (like every other
place that generates an MOSDMap message).

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: avoid useless pg gets when pool is deleted

If the .0 pg no longer exists, we know the entire pool was
deleted, and can avoid querying every other pg.  (This is a good
thing because leveldb and rocksdb can be very slow to query
missing keys.)

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: revamp how pg creates are tracked

Previously we were calculating and managing in-core state that
wasn't committed as part of the pg_map, leading to all sorts of
ugliness that didn't really work.  Instead,

 * set mapping in all creating pgs in the committed pg_map
 * make all pg create message sending be based on committed state
 * update mappings for creating pgs every time we consume a new
   osdmap, so that we have a reliable/stable epoch to attach to
   it.

In particular, having that stable epoch means we have a reference
we can put in the pg create message that will also be used for
the subscription version.  That way OSDs get consistent creates
from any mon.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: only send pg create messages to up osds

If the OSD is down it will ignore the message.  If it gets marked up, we
will eventually consume that map and call check_subs().

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: only churn mapping_epoch if the primary changes

This results is fewer resent pg create messages.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: a bunch of cosmetic cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: drop old creating_pgs_by_osd

Obsoleted by creating_pgs_by_osd_epoch.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: reduce mon_subscribe messages

1. MonClient remembers our subscriptions; only indicate we want
osd_pg_creates once, in init.

2. We don't need to re-request the latest osdmap each time we
reconnect.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: only send new subscriptions

Instead of resending all subscriptions, only send the new ones.  This
avoids races like

 - ask for 4+
 - mon sends maps 4-50
 - ask for 4+ and something else
 - mon has to resend same maps and the other thing

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: send pg creates via persistent subscriptions, not spam

Generate and send pg create messages only for those OSDs who have
subscribed on this monitor.  This is N time more efficient (where there
are N monitors) than the previous method.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: only map and send pg creates post paxos update

These other call sites are no longer needed.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: remove map_pg_creates, send_pg_creates commands

These shouldn't be triggered manually.

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDPGCreate: make it more readable

1- include the epoch
2- drop the 'pg'
3- hide the timestamp

Signed-off-by: Sage Weil <sage@redhat.com>

osd: subscribe to all pg creates, not just once on start

We want to know about all future pg creations, not just those pending
when we start.  (This only helps once the mon knows how to do this...)

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: track creating_pgs_by_osd_epoch

Track pg creations, grouped by the first epoch they mapped to a particular
OSD.  This will be necessary to send messages only for new creations.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: assert our pg counts don't go negative

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: do not prime pg_temp for creating pgs

It will be less work for the old primary to ignore the create message
and the new one to query it and find nothing that for the slightly more
complicated peering and removal process to happen.  Also, this reduces
bloat in the OSDMap a bit.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: note mapping_epoch for creating pgs

Signed-off-by: Sage Weil <sage@redhat.com>

mon: let peon mons send the osdmap replies

Currently the leader mon often replies to OSDs by sending a set of
incremental OSDmaps (e.g., in response to an osd boot or failure).

Instead, send a small message to the proxying peon mon (if any)
with the epoch to start from and let *them* generate a suitable
reply.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/simple/Pipe: show keepalives at level 2

Signed-off-by: Sage Weil <sage@redhat.com>

mon: set mon_subscribe_interval to a day

This is only needed for legacy clients to avoid confusing them--
we don't actually need the renewals at all.  Make them infrequent
to reduce mon load.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: only ack subscriptions (and renew) if client or mon is old

Old client expect an ack so they can schedule renewal; send it for
them only.

Old mons expect renewals.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: remove old subscribe renewal-based timeouts

This is no longer needed/used.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: small cleanup in _ms_dispatch

Signed-off-by: Sage Weil <sage@redhat.com>

mon: new session_timeout mechanism that is not subscribe-based

Simplify the session liveness detection:

 - renew on any message
 - renew on keepalive[2] messages (lightweight ping in msgr)

Signed-off-by: Sage Weil <sage@redhat.com>

msg: make last_keepalive[_ack] lock safe

Signed-off-by: Sage Weil <sage@redhat.com>