osd: remove get_mapped_pools command

No in-tree users.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move ShardedOpWQ::ShardData -> OSDShard

Soon we will destroy pg_map!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: kill _open_lock_pg

Move lock call to caller.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: kill _create_lock_pg

Unused.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: do not release recovery_ops_reserved on requeue

This doesn't make sense.. although it's the same behavior as
luminous.

The point of the releases here is that if we drop something that is in
the queue we drop the recovery_ops_reserved counter by that much.  However,
if something is in the queue and waiting, and we wake it back up, there
is no net change to _reserved... which is only decremented when we
actually dequeue something.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: debug recovery_ops_reserved

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move PG peering waiters into op wq

This resolves problems with a peering event being delivered triggering
advance_pg which triggers a requeue of waiting events that are requeued
*behind* the event we are processing.  It also reduces the number of
wait lists by one, yay!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: store ec profile with final pool

We need this to reinstantiate semi-deleted ec backends.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: ignore RecoveryDone in ReplicaActive too

This can be missed on a RepRecovering -> RepNotRecovering ->
RepWaitBackfillReserved transition.  Catch any straggler events in
ReplicaActive.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: include epoch_sent in pg_query_t operator<<

Signed-off-by: Sage Weil <sage@redhat.com>

osd: restructure pg waiting more

Wait by epoch.  This is less kludgey than before!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: restructure pg waiting

Rethink the way we wait for PGs.  We need to order peering events relative to
each other; keep them in a separate queue in the pg_slot.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: normal command uses slow dispatch (it can send messages)

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSD,PG: get_osdmap()->get_epoch() -> get_osdmap_epoch()

Avoid wrangling shared_ptr!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: misc fixes

Signed-off-by: Sage Weil <sage@redhat.com>

osd: kill disk_tp, recovery_gen_wq

Progress!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move recovery contexts to normal wq

We have a specific PGRecoveryContext type/event--even though we are just
calling a GenContext--so that we can distinguish the event type properly.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove _ookup_lock_pg_with_map_lock_held()

Signed-off-by: Sage Weil <sage@redhat.com>

osd: new MOSDScrub2 message with spg_t, fast dispatch

Send new message to mimic+ OSDs.  Fast dispatch it at the OSD.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: request scrub via a state machine event

Continuing effort to make PG interactions event based.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use peering events for forced recovery

The mgr code is updated to send spg_t's instead of pg_t's (and is slightly
refactored/cleaned).

The PG events are added to the Primary state, unless we're also in the
Clean substate, in which case they are ignored.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: get_primary_shart() variant that returns primary *and* shard

Signed-off-by: Sage Weil <sage@redhat.com>

osd: prime pg_slots for to-be-split children

Once we know which PGs are about to be created, we instantiate their
pg_slot and mark them waiting_pg, which blocks all incoming events until
the split completes, the PG is installed, and we call wake_pg_waiters().

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove obsolete slow dispatch path for most messages

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fast dispatch M[Mon]Command

These just get dumped onto a work queue.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fast dispatch ping

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMOnitor: send MOSDPGCreate2 to mimic+ osds

Signed-off-by: Sage Weil <sage@redhat.com>

osd: handle MOSDPGCreate2 messages (fast dispatch!)

Add a new MOSDPGCreate2 message that sends the spg_t (not just pg_t) and
includes only the info we need.  Fast dispatch it.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMapMapping: a getter that returns a spg_t

Note whether a pool is erasure so that we can generate an appropriate
spg_t for a mapping.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: send pg creations through normal pg queue

Queue a null event tagged with create_info, elimiating the special
legacy path.

These are still not fast dispatch because we need an spg (not pg) to queue
and event, and we need a current osdmap in order to calculate that.  That
isn't possible/a good idea in fast dispatch.  In a subsequent patch we'll
create a new pg create message that includes the correct information and
can be fast dispatched, allowing this path to die off post-nautilus.

Also, improve things so that we ack the pg creation only after the PG has
gone active, meaning it is fully replicated (by at least min_size PGs).

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fix max pg check for peer events

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use atomic for pg_map_size

This avoids the need for pg_map_lock in the max pg check.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PGPeeringEvent: note mon- vs peer-initiated pg creates

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fast dispatch peering events (part 2)

This actually puts the remaining peering events into fast dispatch.  The
only remaining event is the pg create from the mon.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fast dispatch peering events (part 1)

This is a big commit that lays out the infrastructure changes to fast
dispatch the remaining peering events.  It's hard to separate it all out
so this probably doesn't quite build; it's just easier to review as a
separate patch.

- lock ordering for pg_map has changed:
  before:
    OSD::pg_map_lock
      PG::lock
        ShardData::lock

  after:
    PG::lock
      ShardData::lock
        OSD::pg_map_lock

- queue items are now annotated with whether they can proceed without a
pg at all (e.g., query) or can instantiate a pg (e.g., notify log etc).

- There is some wonkiness around getting the initial Initialize event to
a newly-created PG.  I don't love it but it gets the job done for now.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: queue null events without PG lock

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move part of wake_pg_waiters into helper

We'll need this shortly.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use MTrim peering event for trimming

This is simpler and cleaner than handling log trimming as a special case.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fast dispatch backfill and recovery reservation events

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move M{Backfill,Recovery}Reserve event logic into message

Better encapsulation!

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDPeeringOp: add

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: move peering event type out of PG class

We will create these directly from peering Messages shortly.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: keep epoch, not map ref, of last osdmap for lsat persisted epoch

No need to pin the map in memory!

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: remove old update_store_on_load()

This isn't needed post-luminous.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge tag 'v13.0.2'

v13.0.2

Merge PR #21180 into master

* refs/pull/21180/head:
vstart_runner: examine check_status before error

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20460 from colletj/v1_image_creation_disallow

librbd: disallow creation of v1 image format

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21202 from tchaikov/wip-rbd-replay

rbd-replay: remove boost dependency

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21142 from dragonylffly/wip-fix-ebusy

rbd-nbd: fix ebusy when do map

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21157 from trociny/wip-23526

journal: limit number of appends sent in one librados op

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-replay: remove boost dependency

quite a few facilies are now available in standard library now after we
switched to C++17.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #20855 into master

* refs/pull/20855/head:
client: add the fuse parameter max_write

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21096 into master

* refs/pull/21096/head:
pybind/cephfs: added comments to cephfs.pyx

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21183 from neha-ojha/wip-minor-fix-perf-suite

qa/suites/perf-basic: add desc regarding test machines

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #21075 from ceph/wip-yuriw-try-crontab

test:qa:infra - teuthology crontab items as of 3/27/18

qa/suites/perf-basic: add desc regarding test machines

Signed-off-by: Neha Ojha <nojha@redhat.com>

journal: limit number of appends sent in one librados op

Fixes: http://tracker.ceph.com/issues/23526
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21165 from Songweibin/wip-drop-redefine-func

librbd: minor code cleanup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

vstart_runner: examine check_status before error

Issue found by Rishabh Dave. Fix suggested by John Spray.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

CMakeLists: v13.0.2

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21143 from wjwithagen/wjw-ipaddr-ipv6

common: FreeBSD wants the correct struct selection for ipv6

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21136 from kungf/io_events

bluestore: fix aio_t::rval  type

Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20820 from guzhongyan/nodels-improve

mon: node ls improvement

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20298 from markhpc/wip-osd-check_full

osd: don't process ostream strings when not debugging.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #21159 from tchaikov/wip-no-more-use-wheel

src: s/pip --use-wheel/pip/

Reviewed-by: Nathan Cutler <ncutler@suse.com>

pybind/rbd: normalize errno exception

The code meant to do so but actually it does not ...

Signed-off-by: songweibin <song.weibin@zte.com.cn>

librbd: return error when r < 0

Signed-off-by: songweibin <song.weibin@zte.com.cn>

librbd: drop a duplicate definition function

Signed-off-by: songweibin <song.weibin@zte.com.cn>

src: s/--use-wheel//

--use-wheel was deprecated in favor of --only-binary in pip v7.0.0. and
--use-wheel was removed in a recent release of pip. but some packages
are source packages, so we cannot simply replace use-wheel with
only-binary. so a simpler approach is to drop --use-wheel option, as pip
respects --find-links, and will find the required package from the
wheelhouse.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #20132 into master

* refs/pull/20132/head:
qa/cephfs: update TestDamage for open file table
mds: allow storing open file table in multiple omaps
mds: differentiate Anchor types to clarify purpose
mds: add perf counter for 'open ino' operation
mds: protect open file table against partial omap update
mds: add dirfrags whose child inodes have caps to open file table
mds: don't try prefetching destroyed inodes
mds: don't try opening inodes that haven't been created
mds: don't re-requeue open files to head of log
mds: use open file table to speed up mds recovery
mds: introduce open file table
mds: track how many clients/mds want caps for each inode
mds: cleanup MDCache::opening_inodes access
mds: cleanup CInode/CDir states definition

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21147 from rjfd/wip-dashboard-fix-notification-queue-test

mgr/dashboard: notification queue: fix priority tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: notification queue: fix priority tests

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #20979 from ceph/wip-cd-smoke-fix

qa/tests: use ceph-deploy stable branch for single node tests

qa/tests: use stable branch for single node tests

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

common: FreeBSD wants the correct struct selection for ipv6

Lets see if this also works for Linux

Fixes: http://tracker.ceph.com/issues/21813
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rbd-nbd: fix ebusy when do map

When doing rbd-nbd map, if the Ceph service is not available,
the codes will wait on rados.connect(), unless killing the process.
In that case, the close_nbd logic is skipped with NBD_CLEAR_SOCK ioctl
not called. On the CentOS 7 kernel, it leaves nbd->file not cleared, which
causes the subsequent map requests return EBUSY, this patch fixes it
by connecting Ceph first prior to calling NBD_SET_SOCK ioctl

Fixes: http://tracker.ceph.com/issues/23528
Signed-off-by: Li Wang <laurence.liwang@gmail.com>

librbd/cli-integration: Permit V1 image creation

Signed-off-by: Julien COLLET <julien.collet@cern.ch>

Merge pull request #18317 from hitoshikamei/master-rbd

rbd: eager-thick provisioning support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20868 from shun-s/fix-same-cookie

librbd: fix refuse to release lock when cookie is the same at rewatch

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20935 from trociny/wip-get-name

librbd: add API function to get image name

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: change the type of aio_t:res to long

io_event::res was assigned to io_t::rval, but io_event::res
is long, if rval is int, this will cause overflow.

Fixes: http://tracker.ceph.com/issues/23527
Signed-off-by: kungf <yang.wang@easystack.cn>

Merge pull request #21117 from smithfarm/wip-23504

tests: remove TestPGLog ASSERT_DEATH test

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #20927 into master

* refs/pull/20927/head:
doc: use actual entity address for clarity
doc: make minor grammatical rectifications

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21131 from trociny/wip-23502

qa/suites/rbd: set qemu task time_wait param

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd/OSD: make dout in _check_full conditional

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge PR #20983 into master

* refs/pull/20983/head:
ceph_authtool: add mode option

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph_authtool: add mode option

We now have the ability to set the keyring file mode with the help of
'--mode MODE'. The mode needs to be specified in octal using the format:
0600.

Closes: http://tracker.ceph.com/issues/23513
Signed-off-by: Sébastien Han <seb@redhat.com>

qa/suites/rbd: set qemu task time_wait param

so workloads qemu_dynamic_features.sh and qemu_rebuild_object_map.sh,
which check if qemu is finished with periodicity 60 sec, have enough
time to detect this before the rbd image is removed.

Fixes: https://tracker.ceph.com/issues/23502
Signed-off-by: Mykola Golub <mgolub@suse.com>

qa/tasks/qemu: add a parameter to wait for workloads detect qemu finished

In the case when a workload needs to detect qemu finished by running a
check with a periodicity of N sec it needs to set time_wait to 2 * N
in order to avoid races on finish.

Signed-off-by: Mykola Golub <mgolub@suse.com>

test:qa:infra - teuthology crontab items as of 3/27/18

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #21121 from tchaikov/wip-no-relicensing

msg/async/dpdk: remove xsky copyright and LGPL copying

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #21105 from ashitakasam/wip-bug

rgw: comparison between signed and unsigned integer expressions

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21074 from trociny/wip-librados_test_stub-append

test/librados_test_stub: always create copy of buffers passed to operation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21116 from trociny/wip-23499

qa/workunits/rbd: remove sanity check in test_admin_socket.sh

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

msg/async/dpdk: remove xsky copyright and LGPL copying

it's not legitimate to relicense source code without original author's
permission. some of the source files are copied from seastar project
with minor changes. so xsky cannot claim the copyright of them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21107 from tchaikov/wip-pg-stat-cap-trimq

osd: cap snaptrimq_len at 2^32

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #21106 from tchaikov/wip-compressor-zstd

compressor: add zstd back

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21097 from ceph/wip-ifed-fix-reclaim-cap-bug

os/bluestore: fix broken cap in _balance_bluefs_freespace()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20556 from yanghonggang/master

lttng: Trace rgw data transfer, bi entry and object header update processes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

tests: remove TestPGLog ASSERT_DEATH test

Fixes: http://tracker.ceph.com/issues/23504
Signed-off-by: Nathan Cutler <ncutler@suse.com>