Merge pull request #15398 from liewegas/wip-bluestore-leak

os/bluestore: fix buffers pinned by indefinitely deferred writes

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15402 from liewegas/wip-bluestore-mempools

os/bluestore: improve mempool usage

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15403 from liewegas/wip-mempool-dump

mempool: improve dump; fix buffer accounting bugs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15431 from rzarzynski/wip-rgw-vim-modeline-for-orhpantool

rgw: add the Vim's modeline into rgw_orphan.cc.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15425 from yaozongyou/fix-build-error

cmake: link against fcgi only if enabled

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: add the Vim's modeline into rgw_orphan.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #15408 from yonghengdexin735/wip-zzz-add_const

include/lru.h: add const to member functions

Reviewed-by: Jos Collin <jcollin@redhat.com>

Fix cmake build error.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #15420 from yuriw/wip_remove_default_idle_timeout

qa: Removed all 'default_idle_timeout' due to chnage in rwg task

Merge pull request #15397 from dzafman/wip-20089

osd: Object level shard errors are tracked and used if no auth available

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15417 from smithfarm/wip-cond-epoch-2

build/ops: rpm: move _epoch_prefix below Epoch definition

include/lru: add const to member function

Signed-off-by: yonghengdexin735 <zhang.zezhu@zte.com.cn>

Removed all 'default_idle_timeout' due to chnage in rwg task
https://github.com/ceph/ceph/commit/8c74c8a639cd3dead52e1942b36f6eb3f1ceab2c#diff-995b04809fcabacc3e3ecfaea903a41aL539

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

build/ops: rpm: move _epoch_prefix below Epoch definition

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #15400 from yehudasa/wip-20128

qa/rgw_snaps: move default_idle_timeout config under the client

osd: Object level shard errors are tracked and used if no auth available

Shards with object mismatch are tracked to mark them inconsistent
Fix test because storing omap_digest in object_info not behaving as before

Fixes: http://tracker.ceph.com/issues/20089
Signed-off-by: David Zafman <dzafman@redhat.com>

osd-scrub-repair: Fix broken test and correctly handle ec overwrite testing

Caused by: 3ca750d41dfe33c6efea4abc96d2bd426a9742b9

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #15212 from neha-ojha/wip-fix-recovery-sleep-configurable

osd: Implement asynchronous recovery sleep

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15262 from Adirl/revert_cm

msg/async: Revert RDMA-CM

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15307 from ovh/bp-64bit-mem-is-zero

inline_memory: optimized mem_is_zero for non-x64

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15340 from wjwithagen/wip-wjw-libcommon

cmake: do not link libcommon against some libs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15342 from chardan/jfw-wip-add_VLA_warning

build: Adds C++ warning flag for C Variable-Length Arrays.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15364 from joscollin/wip-cleanup-putto-operator-const

src: put-to operator function - const input cleanup

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15372 from ganeshmaharaj/reuse-isal

erasure-code: Remove duplicate of isa-l files

Reviewed-by: Loic Dachary <ldachary@redhat.com>

os/bluestore: release aios and pinned buffers on io complete

Once we're done with our IO, clear the aio list so that the pinned buffers
are unpinned.  This ensures we release memory quickly, even if the
TransContext sticks around for a while (e.g., in the osr q).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: submit deferred if txc cleanup is blocked

If we have a single deferred write, and then a uniform workload with *no*
deferred writes, we will never actually submit it.  Meanwhile, the txc is
stuck on the osr q and nothing ever gets retired.

Simple fix is to submit any deferred ops if the osr queue is blocked by
a queued deferred write and the osr queue length is above some
threshold.  This prevents memory from being pinned indefinitely.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15209 from liewegas/wip-sortbitwise-notes

doc/release-notes: update which jewel version does sortbitwise warning

Merge pull request #15274 from liewegas/wip-auto-ccache

do_cmake.sh: enable ccache if installed

Reviewed-by: Kefu Chai <kchai@redhat.com>

do_cmake.sh: enable ccache if installed

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15391 from liewegas/wip-authors

AUTHORS: update with release manager, backport team

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #15410 from dachary/wip-19941-ceph-disk

ceph-disk: do not setup_statedir on trigger

Reviewed-by: David Disseldorp <ddiss@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15412 from dachary/wip-choose-args

crush: add missing tunable in tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

crush: add missing tunable in tests

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #15286 from smithfarm/wip-cond-epoch

build/ops: rpm: apply epoch only if %epoch macro is defined

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15240 from ukernel/wip-20055

osdc/Journaler: avoid executing on_safe contexts prematurely

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15282 from david-z/wip-fix-mantle

mds: Pass empty string to clear mantle balancer

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15314 from renhwztetecs/renhw-wip-client-cleanup

client: fix display ino in the ldout

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #15300 from ukernel/wip-20083

mds: properly create aux subtrees for pinned directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15278 from viCEnTECheNg/wip-data-pools-type

mds: change the type of data_pools

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15205 from batrick/i20039

mds: check export pin during replay

Reviewed-by: Yan, Zheng <zyan@redhat.com>

ceph-disk: do not setup_statedir on trigger

trigger may run when statedir is unavailable and does not use it.

Fixes: http://tracker.ceph.com/issues/19941
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #15406 from tchaikov/wip-setuptools-vendors-six-or-not

install-deps.sh: workaround setuptools' dependency on six

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

install-deps.sh: workaround setuptools' dependency on six

see also https://github.com/pypa/setuptools/issues/1042

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: mempools for writing and writing_deferred

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: bluestore_txc mempool

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: rename cache mempools to bluestore_cache_*

These map directly onto the bluestore_cache_size setting.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: maybe_rebuild before adding to cache too

Signed-off-by: Sage Weil <sage@redhat.com>

AUTHORS: update with release manager, backport team

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15370 from liewegas/wip-jewel-tunables

crush, mon: make jewel the lower bound for client/crush compat for new clusters

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test/cli/osdmaptool/: fix tests

CRUSH tunables now default to jewel; outputs vary and
upmap results are a bit different.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: fix some append_buffer vs mempool assignment; add tests

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: make dump more concise

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/crushtool: adjust tests

Specify chooseleaf_stable for --build tests to match prior output.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: set mon_osd_initial_require_min_compat_client = jewel

IMO a jewel lower bound is reasonable for new clusters.

Signed-off-by: Sage Weil <sage@redhat.com>

crush: set default tunables to jewel

For new clusters.

This is the last "disruptive" tunable change we've made (where adjusting/
fixing it requires lots of data movement).  For new clusters a lower bound
of jewel seems pretty reasonable, and this has been upstream in Linux
for a while now.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15267 from joscollin/wip-cleanup-redundant-headers-5

common: Remove redundant includes - 5

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

qa/rgw_snaps: move default_idle_timeout config under the client

Fixes: http://tracker.ceph.com/issues/20128
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15311 from dachary/wip-crush-add-remove

crush: update choose_args when items are added/removed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15318 from tchaikov/wip-20067

mon: trim the creating_pgs after updating it with pgmap

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15352 from liewegas/wip-buffer-mempools

buffer: allow buffers to be accounted in arbitrary mempools

Merge pull request #15357 from ceph/typo_bluestore

os/bluestore: fix a typo about bleustore

Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #15385 from myoungwon/wip-20114

test/osd/TestRados.cc: run set-redirect test after finishing setup

Reviewed-by: Sage Weil <sage@redhat.com>

Revert "osd: When scrub finds an attr error mark shard inconsistent"

This reverts commit 44736aac74493b5532c9e3574a37a3642766f212.

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #14457 from ceph/wip-rest-flask

ceph-mgr: Implement new pecan-based rest api

Merge pull request #15222 from guihecheng/rgw_file-fix-unlink

rgw_file: fix flags set on unsuccessful unlink

Merge pull request #15329 from guihecheng/rgw_file-readdir

rgw_file: add lock protection for readdir against gc

Merge pull request #14871 from ceph/wip-selinux-optimize

selinux: Do parallel relabel on package install

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

mempool: dump total, too

Signed-off-by: Sage Weil <sage@redhat.com>

osd: (loosely) mark cached map bl buffers as osd_mapbl

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: mark buffer caches buffers as bluestore_data

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: remove buffer_data mempool

We are explicitly accounting for buffers via arbitrary pools, defaulting
to buffer_anon, so we don't need buffer_data anymore.  It leads to
double-counting.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: allow buffers to map into arbitrary mempools

- default is buffer_anon
- buffer_data includes *all* buffers across all mempools, and is the
*actual* allocation (includes raw_combined).
- buffer_meta is other buffer-related overhead.

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: allow pool counts to be adjusted explicitly

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: fix mempool accounting for claimed chars

These are allocated by the caller, outside of the mempool; do not
deallocate via the mempool.

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: kill unittest_? mempools

These were just annoying.  Use some real ones that will be empty
for the unit tests.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15291 from liewegas/wip-osdmap-pgtemp

osd/OSDMap: make pg_temp more efficient

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15321 from liewegas/wip-failed-msg

mon/OSDMonitor: change info in 'osd failed' messages

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15388 from idryomov/wip-krbd-fallocate-tests

qa: krbd discard/zeroout tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15135 from cbodley/wip-denc-sstring

denc: add encode/decode for basic_sstring

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13802 from LiumxNL/wip-170306

mon/OSDMonitor: use up set instead of acting set in reweight_by_utilization

Reviewed-by: Kefu Chai <kchai@redhat.com>

erasure-code: Remove duplicate of isa-l files

There are two copies of isa-l. While one is a git submodule the other
was a static copy from an older version. This change helps with removing
the duplicate and maintain a single copy.
This is also upgrading isa-l used by (src/erasure-code/isa/isa-l) from
v2.14 to v2.16.0

Signed-off-by: Ganesh Mahalingam <ganesh.mahalingam@intel.com>

Merge pull request #15295 from liewegas/wip-bluestore-fsck

os/bluestore: fix fsck deferred_replay

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

test/osd/TestRados.cc: run test after finishing redirect setup

set-redirect test creates more objects than default due to redirect mapping
(let's assume that we create 500 objects, oid 1 -> oid 501, oid 2 -> oid 502)
But the test should use objects within pre-defined range. For example,
If the number of object is 500, only 1~500 oid are allowed for test operations.

In the failure cases, test operation tried to use wrong oids (which means
higher than 500). This is because the bug in the setup phase.
If max_inflights > 1, set-redirect can be finished after setup is completed.
(so wrong oid is included in oid_in_use).
We need to completely remove redirect objects in oid_in_use before start the test.

Fixes: http://tracker.ceph.com/issues/20114
Signed-off-by: Myoungwon Oh <omwmw@sk.com>

qa: krbd discard/zeroout tests

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rpm: Move ceph-disk to ceph-base

The SELinux package now requires the ceph-disk binary but that one was
part of the ceph-osd package. The ceph-disk python library is already
packaged in ceph-base so moving ceph-disk to ceph-base seems like a
reasonable next step.

Signed-off-by: Boris Ranto <branto@redhat.com>

rgw_file: add lock protection for readdir against gc

The state.mtx protects state.events accessed both from readdir and gc.
Fixes: http://tracker.ceph.com/issues/20121
Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>

Merge pull request #15374 from yonghengdexin735/wip-zzz-mds

mon: Removed unnecessary function declaration in MDSMonitor.h

Reviewed-by: Jos Collin <jcollin@redhat.com>

crush: update choose_args on adjust_item_weight

The osd_crush_update_weight_set (true by default) can be used to disable
the update of the weights.

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: update choose_args when items are added/removed

When items are added:

- the value in the weight set is set to the target weight. It is assumed
  that weight set are updated on a regular basis and will eventually be
  set to a value that prevents excessive over/under filling.

- the value in the id list is set to the item id.

When items are removed, their weight / ids in all choose_args are
removed.

Signed-off-by: Loic Dachary <loic@dachary.org>

src: put-to operator function - const input cleanup

Fixed the instances of the issue specified in Tracker#3977 inside src/ folder utmost, by adding
const input. This fix avoids the following directories inside src: rocksdb/ zstd/ boost/ rapidjson/ googletest/ Beast/.

Fixes: http://tracker.ceph.com/issues/3977
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15373 from kestrels/wip-asm-capital-S

build: Use .S suffix for ppc64le assembly files

Reviewed-by: Bassam Tabbara <bassam.tabbara@quantum.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

mon:remove useless function for MDSMonitor::generate_mds_map()

Signed-off-by: yonghengdexin735 <zhang.zezhu@zte.com.cn>

Merge pull request #15351 from liewegas/wip-osdmap-require-min-client

osd/OSDMap: replace string-based min_compat_client with a CEPH_RELEASE_* uint8_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osdmaptool: fix cli tests

Signed-off-by: Sage Weil <sage@redhat.com>

common/version: include release name and type in version string

 ceph version 12.0.2-1687-ge042051 (e0420518d712d87a4882b3a327eaf1d3fcea12df) luminous (dev)

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: switch [require_]min_compat_client to CEPH_RELEASE_* values

Instead of using a string, use the defined CEPH_RELEASE_* values.  This
is simpler.

We allow decoding of post-kraken pre-luminous maps that use the string
values.

Signed-off-by: Sage Weil <sage@redhat.com>

common: expand CEPH_RELEASE_ names; rename helper

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15331 from yonghengdexin735/wip-zzz-override

mon: Add override for FsNewHandler::handle()

Reviewed-by: Jos Collin <jcollin@redhat.com>

build: Use .S suffix for ppc64le assembly files

The assembly file name suffix was changed from .S to .s,
which caused a build break on ppc64le architecture.

To fix this issue, the suffix will be changed back to .S
for ppc64le assembly files only.  The x86 assembly files
can retain the .s suffix.

Fixes: http://tracker.ceph.com/issues/20106
Signed-off-by: Andrew Solomon <asolomon@us.ibm.com>

mon: Add override for FsNewHandler::handle()

Signed-off-by: yonghengdexin735 <zhang.zezhu@zte.com.cn>