doc: mention ENXIO change in the 10.2.6 release notes

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13880 from cbodley/wip-qa-submodule

build: remove ceph-qa-suite directory

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #13217 from liewegas/wip-bluestore-prefer-wal-size

os/bluestore: add bluestore_prefer_wal_size option

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

build: remove ceph-qa-suite directory

after commit 3afb889 "qa: add supported distros for ceph-ansible", git
submodule update commands are failing with:

No submodule mapping found in .gitmodules for path 'ceph-qa-suite'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13785 from liewegas/wip-cleanup-features

msg: no need to pass supported features to Messenger::Policy ctor

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13078 from zhangsw/fix-rgw-setacl-bug

rgw: put object's acl can't work well on the latest object

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13852 from liewegas/wip-19133

qa/suites: fix upgrade tests vs cluster full thrashing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13711 from tmuthamizhan/wip-add-supported-distros

qa: add supported distros for ceph-ansible

Reviewed-by: Yuri Weinstein <yweinstei@redhat.com>

Merge pull request #13776 from Liuchang0812/wip-add-override-in-tool-headers

tools: add override in tool submodule

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13791 from runsisi/wip-rbd-ack-cleanup

librbd: rbd ack cleanup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13775 from Liuchang0812/wip-add-override-librados-headers

librados: add override in headers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13774 from Liuchang0812/wip-add-override-in-common-headers

common: add override in header file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13773 from Liuchang0812/wip-add-override-in-test-headers

test: add override in test submodule

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13771 from Liuchang0812/wip-add-override-in-msg-headers

common: add override in msg subsystem

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13782 from vshankar/wip-19034

rbd: use concurrent writes for imports

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13752 from trociny/wip-19130

librbd: relax "is parent mirrored" check when enabling mirroring for pool

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13444 from dachary/wip-18943-device-class

crush: add devices class that rules can use as a filter

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13835 from theanalyst/doc/releases/10.2.6

doc: Jewel 10.2.6 release notes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13453 from xiaoxichen/drop_mallinfo

common/MemoryModel: Bump int to long and drop mallinfo

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13580 from ukernel/wip-18914

client: wait for lastest osdmap when handling set file/dir layout

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13691 from Liuchang0812/wip-add-override-in-mds-headers

mds: add override in headers

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #12786 from jcsp/wip-11950

#11950: Persistent purge queue

Reviewed-by: Yan, Zheng <zyan@redhat.com>

mds: handle Journaler::recover errors in PurgeQueue

Signed-off-by: John Spray <john.spray@redhat.com>

qa: update TestDamage for PurgeQueue

Signed-off-by: John Spray <john.spray@redhat.com>

qa: update TestFlush for changed stray perf counters

Signed-off-by: John Spray <john.spray@redhat.com>

mds: flush PQ even when not consuming

In normal operation we generate flushes from
_consume when we read from the journaler.  However,
we should also have a fallback flush mechanism for
situations where can_consume() is false fo a long time.

This comes up in testing when we set throttle to zero to
prevent progress, but would also come up in real life if
we were busy purging a few very large files, or if purging
was stuck due to bad PGs in the data pool -- we don't want
that to stop us completing appends to the PQ.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: add TestStrays.test_purge_queue_op_rate

For ensuring that the PurgeQueue code is not generating
too many extra IOs.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: write_head when reading in PurgeQueue

Previously write_head calls were only generated
on the write side, so if you had a big queue
and were just working through consuming it, you
wouldn't record your progress, and on a daemon
restart would end up repeating a load of work.

Signed-off-by: John Spray <john.spray@redhat.com>

osdc: expose Journaler::write_head_needed

So that callers on the read side can optionally
do their own write_head calls according to
the same condition that Journaler uses
internally for its write_head during _flush() condition.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: remove unnecessary flush() from PurgeQueue

We can drive all flushing from the read side.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: update for removing Timer from Journaler

Signed-off-by: John Spray <john.spray@redhat.com>

osdc: less aggressive prefetch in read/write Journaler

Previously, if doing a write/is_readable/write/is_readable sequence,
you'd end up doing a flush after every write, even though there
was already a flush in flight that would advance the readable-ness
of the journal.

Because this flush-during-read path is only active when using
a read/write journal such as in PurgeQueue, tweak the behaviour
to suit this case.

Signed-off-by: John Spray <john.spray@redhat.com>

osdc: remove Journaler "journaler_batch_*" settings

This was an unused code path.  If anyone set a nonzero
value here the MDS would crash because the Timer implementation
has changed since this code was written, and now requires
add_event_after callers to hold the right lock.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add error handling in PurgeQueue

For decode errors, and for Journaler errors.
Both are considered damage to the MDS rank, as
with other per-rank data structures.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: expose progress during PurgeQueue drain

We don't track an item count, but we do have
a number of bytes left in the Journaler, so
can use that to give an indication of progress
while the MDS rank shutdown is waiting for
the PurgeQueue to do its thing.

Also lift the ops limit on the PurgeQueue
when it goes into the drain phase.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: update PurgeQueue for single-ack OSD change

Signed-off-by: John Spray <john.spray@redhat.com>

mds: create purge queue if it's not found

Signed-off-by: John Spray <john.spray@redhat.com>

qa/cephfs: add TestStrays.test_purge_on_shutdown

...and change test_migration_on_shutdown to
specifically target non-purgeable strays (i.e.
hardlink-ish things).

Signed-off-by: John Spray <john.spray@redhat.com>

mds: wait for purgequeue on rank shutdown

Also, move shutdown_pass call from dispatch
to tick, so that it doesn't rely on incoming
messages to make progress.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: update test_strays for purgequeue

Signed-off-by: John Spray <john.spray@redhat.com>

mds: implement PurgeQueue throttling

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add stats to PurgeQueue

Signed-off-by: John Spray <john.spray@redhat.com>

mds: move dir purge and truncate into purgequeue

Signed-off-by: John Spray <john.spray@redhat.com>

mds: move throttling code out of StrayManager

This will belong in PurgeQueue from now on.  We assume
that there is no need to throttle the rate of insertions
into purge queue as it is an efficient sequentially-written
journal.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: move PurgeQueue up to MDSRank

To better reflect its lifecycle: it has a part to play
in create/open and has an init/shutdown, unlike StrayManager.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: use a persistent queue for purging deleted files

To avoid creating stray directories of unbounded size
and all the associated pain, use a more appropriate
datastructure to store a FIFO of inodes that need
purging.

Fixes: http://tracker.ceph.com/issues/11950
Signed-off-by: John Spray <john.spray@redhat.com>

osdc/Journaler: wrap recover() completion in finisher

Otherwise, the callback will deadlock if it in turn
calls into any Journaler functions.  Don't care
about performance because we do this once at startup.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: const snaprealm getters on CInode

Signed-off-by: John Spray <john.spray@redhat.com>

mds: const methods on SnapRealm

Signed-off-by: John Spray <john.spray@redhat.com>

osdc/Filer: const fix for passed layouts

...so that const references can be passed into
purge calls.

Signed-off-by: John Spray <john.spray@redhat.com>

common/lockdep: clearer log messages

Previously these were contextless "using id..." messages with
no indication of what subsystem the message came from.

Signed-off-by: John Spray <john.spray@redhat.com>

osdc/Journaler: add have_waiter()

Allows users of wait_for_readable to conveniently
see if there is already a waiter.  Yes, they could
do this themselves, but I'd rather peek at an existing
variable than add a new one caller-side.

Signed-off-by: John Spray <john.spray@redhat.com>

osdc/Journaler: remove incorrect assertion

This asserted that flush_pos would be ahead of
safe_pos after calling _flush.  However, this
is not guaranteed to be the case because
prezeroing might prevent us from flushing
right now.

Signed-off-by: John Spray <john.spray@redhat.com>

osdc/Journaler: assign a name for logging

Now that we have an MDLog journaler and a PurgeQueue journaler,
this is needed to avoid confusion.

Signed-off-by: John Spray <john.spray@redhat.com>

compact_set: add #includes for dependencies

This was previously working by side effects, I happened
to include it somewhere that its dependencies weren't
already included.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #13397 from SUSE/doc-fix-qa-links

doc: update links to point to ceph/qa instead of ceph-qa-suite

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13816 from batrick/i19201

mds: print rank as int

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #13830 from jcsp/wip-doc-multimds

doc: instructions and guidance for multimds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: Jewel 10.2.6 release notes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: mention interactive task in developer guide

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: rewrite Deploy a cluster for manual testing section

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: update links to point to ceph/qa instead of ceph-qa-suite

Also fix two broken links to install task and two typos.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #12506 from SUSE/wip-18259

Revert "dummy: reduce run time, run user.yaml playbook"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

qa/suites/upgrade/jewel-x: do not thrash cluster full during upgrade

The mon command varies.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suite/rados/upgrade: do not thrash full cluster during upgrade

The mon command varies!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13615 from liewegas/wip-osd-full

mon,osd: new mechanism for managing full and nearfull OSDs for luminous

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #13759 from liewegas/wip-19133

osdc/Objecter: resend RWORDERED ops on full

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #13734 from liewegas/wip-jewel-x

qa/suite/upgrade/jewel-x: various fixes

Reviewed-by: Yuri Weinstein <yweinstei@redhat.com>

rbd: refactor header import

and resuse validate_banner() helper routine where
ever necessary.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

rbd: concurrent v2 image IO during import/import-diff

Fixes: http://tracker.ceph.com/issues/19034
Signed-off-by: Venky Shankar <vshankar@redhat.com>

rbd: cleanup unused throttle in v2 import

v2 import does not use throttle as of now although v1
import does use it - initialize throttle wherever its
necessary and avoid passing it functions that do not
require it.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #13846 from rzarzynski/wip-qa-rgw-start-apache-first

qa/tasks/rgw.py: start Apache before RadosGW.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

os/bluestore: drop unused OpSequencer::wait_for_wal_on_seq()

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add bluestore_prefer_wal_size[_hdd,_ssd] options

Add option to prefer a WAL write if the write is below a size threshold,
even if we could avoid it.  This lets you trade some write-amp (by
journaling data to rocksdb) for latency in cases where the WAL device is
much faster than the main device.

This affects:

 - writes to new extents locations below min_alloc_size
 - writes to unallocated space below min_alloc_size
 - "big" writes above min_alloc_size that are below the prefer_wal_size
   threshold.

Note that it's applied to individual blobs, not the entirety of the write,
so if your have a larger write torn into two pieces/blobs that are below
the threshold then they will both go through the wal.

Set different defaults for HDD and SSD, since this makes more sense for HDD
where seeks are expensive.

Add some test cases to exercise the option.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: use new luminous set-full-ratio etc

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13410 from yehudasa/wip-tracing-fix

tracing: don't include oid when tracing at dequeue_op()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13839 from theanalyst/release/10.2.6/changelog

doc: add changelog for v10.2.6 Jewel release

doc: add changelog for v10.2.6 Jewel release

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #13704 from batrick/mds-counter-unify

mds: remove some redundant object counters

Reviewed-by: Yan, Zheng <zyan@redhat.com>

osdc/Objecter: resend RWORDERED ops on full

Our condition for respecting the FULL flag is complex, and involves
the WRITE | RWORDERED flags vs the FULL_FORCE | FULL_TRY flags.  Previously,
we could block a read bc of RWORDRED but not resend it later.

Fix by capturing the complex condition in a respects_full() bool and using
it both for the blocking-on-send and resending-on-possibly-notfull-later
checks.

Fixes: http://tracker.ceph.com/issues/19133
Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/thrashosds: chance_thrash_cluster_full

Induce a momentarily full cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13832 from linuxbox2/wip-rgw-fs_inst

rgw_file:  fix fs_inst progression

Merge pull request #10240 from songbaisen/b2

mon: remove the redudant jugement in paxosservice is_writeable function

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw_file:  fix fs_inst progression

Reported by Gui Hecheng<guimark@126.com>.  This change is a
variation on proposed fix by Dan Gryniewicz<dang@redhat.com>
to take root_fh.state.dev as fs_inst for new handles.

Fixes: http://tracker.ceph.com/issues/19214
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

qa/tasks/rgw.py: start Apache before RadosGW.

At the end of start_rgw() we wait till establishing HTTP connections
with RadosGW become possible. However, if RadosGW uses the FastCGI,
the condition can't be fulfilled without spawning HTTP server first.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

librbd: relax "is parent mirrored" check when enabling mirroring for pool

If the parent is in the same pool and has the journaling feature enabled
we can assume the mirroring will eventually be enabled for it.

Fixes: http://tracker.ceph.com/issues/19130
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #13323 from yehudasa/wip-18079-2

librados: use cursor for nobjects listing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13262 from batrick/multimds-thrasher

Add multimds:thrash sub-suite and fix bugs in thrasher for multimds

Reviewed-by: John Spray <john.spray@redhat.com>

doc: instructions and guidance for multimds

Inspired by http://tracker.ceph.com/issues/19135

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #13560 from wjwithagen/wip-wjw-ceph-disk-tests

ceph-disk/tests: Certain partition types do not work on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13810 from yuyuyu101/wip-rdma-inflight

msg/async/rdma: destroy QueuePair if needed

Reviewed-by: Adir lev <adirl@mellanox.com>

Merge pull request #13742 from liupan1111/wip-cleanup-journal

os/filestore: use existing variable for same func.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12177 from kylinstorage/wip-remove-unneeded-loop

os/filestore/FileStore.cc: remove unneeded loop

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13741 from rzarzynski/wip-bs-fastcrc32-in-rocks

os/bluestore: enable SSE-assisted CRC32 calculations in RocksDB

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13768 from tchaikov/wip-clang-fixes

librados, osd: clang fixes

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #13794 from liewegas/wip-clog-newlines

common: remove \n on clog messages

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #13796 from liewegas/wip-debian-base-dbg

debian/control: add ceph-base-dbg

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/upgrade/jewel-x/parallel: upgrade mons before osds

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/jewel-x/parallel: expand workload matrix

These should run independently against a racing upgrade.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13799 from optimistyzy/36_1

bluestore, NVMEDEVICE: Specify the max io completion in conf

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Pan Liu <liupan1111@gmail.com>