lockdep: Convert free_ids from list to bitmap

Convert "free_ids" list to bitmap, reducing its memory usage from
16 + MAX_LOCKS * 20 (~20KB) to MAX_LOCKS/8 (512b) and save some time
(re)populating the free id array.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

lockdep: Reduce "follows" matrix size by 8

Convert "follows" matrix to 2-dimensional array of bits, reducing its
storage size from 16 MB (assuming MAX_LOCKS = 4096) to 2MB.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

lockdep: Reduce matrices usage range to minimum

Introduce current_maxid variable which holds highest lock ID. Using that,
don't attempt to check/reset parts of "follows" and "follows_bt" matrices
that are not used yet. This improves performance by reducing CPU usage
(particularly by lockdep_unregister() function), this also reduces RSS
memory usage by quite a bit, as many allocated pages are not actually
touched anymore. Improve CPU usage a bit more, by using memset() to
reset "follows" members instead of iterating byte-by-byte.
Finally, use push_front instead of push_back when unregistering, so
unregistered IDs will be recycled.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

Merge pull request #9241 from ceph/wip-fix-q-tests

common/: fix tests for prioritized_queue and wpq

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #9102 from yehudasa/wip-rgw-leak-4

rgw: don't unregister request if request is not connected to manager

Reviewed-by: Casey Bodley <cbodley@redhat.com>

common/: fix tests for prioritized_queue and wpq

Introduced: 27b6ec40b93c05fbaa228783e11e2fac8f1a1ee1
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #9216 from dzafman/wip-test-fix

test: Disable tests due to recovery race

test: Disable tests due to recovery race

Disable:
osd-scrub-repair.sh
test-erasure-eio.sh
test-erasure-code.sh

These test are failing due to race conditions that is exposing
bug http://tracker.ceph.com/issues/13937.

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #9211 from dillaman/wip-15938

librbd: write-after-write might result in an inconsistent replicated image

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9231 from athanatos/wip-recovery-wq

osd: Put background recovery tasks in the main OpWQ

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6208 from liewegas/wip-alloc-hints

librados,osd: rados hints for future access pattern

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7070 from majianpeng/headerbeatmap-check

common/HeartbeatMap: print stack info of unhealth thread.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7843 from branch-predictor/bp-smart-async-wp

msg/async: Implement smarter worker thread selection

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #8141 from songbaisen/song14

crush: fix the problem can not find the define item below the bucket

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8840 from tchaikov/wip-opt-u32

common/config: cast OPT_U32 options using uint32_t

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9072 from liewegas/wip-15840

osd: fix crush location update on startup

Merge pull request #9116 from ddiss/librados_sparse_read_use_snapid

librados: use caller provided snapid for aio_sparse_read()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #9002 from rzarzynski/wip-rgw-swift-acct_cont_md-fix

rgw: fix updating account/container metadata of Swift API.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9225 from dachary/wip-run-make-check

tests: rm -fr /tmp/*virtualenv*

Reviewed-by: Erwan Velu <erwan@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9165 from mikulely/fix-rgw-multisite-test

rgw: misc fix for test_multi.py

Reviewed-by: Casey Bodley <cbodley@redhat.com>

tests: rm -fr /tmp/*virtualenv*

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #9139 from majianpeng/bluestore-statfs

os/bluestore/BlueStore: free space of bluestore should reduce spaces …

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9161 from cxwshawn/wip-tc1-fix

osd: optimize header file dependency to decreace compile time when modify any file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9193 from cxwshawn/wip-tc2-fix

mon: optimize header file dependency.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9068 from trociny/wip-pybind-rbd-mirror

librbd: initial python APIs to support mirroring

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9088 from trociny/wip-15500

qa: dynamic_features.sh: return error only if it failed on alive QEMU

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9140 from zhouyuan/fix_disable_journaling

librbd: close journal before removing

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9177 from trociny/wip-15916

rbd-mirror: it might got stuck in calculating "behind master" position

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

journal: replay position might change after pruning stale tags

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: delay commit of overwritten journal event

With the cache enabled and write-after-write IOs to the same
object extents, it was possible for the overwritten journal event
to be committed before the overwriter journal event was written
to disk.  If a client crash occurs before the event is written,
the image will be inconsistent on replay.

Fixes: http://tracker.ceph.com/issues/15938
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9196 from tchaikov/wip-cmake-vstart

cmake: add cython_rados to vstart target

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8164 from rzarzynski/wip-rgw-bulkdel-slashfix

rgw: handle initial slashes properly in BulkDelete of Swift API.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: handle initial slashes properly in BulkDelete of Swift API.

Fixes: http://tracker.ceph.com/issues/15948
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #9220 from cxwshawn/wip-tc3-fix

auth: optimize header file dependency.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9212 from liewegas/wip-bit-alloc-test

tests: disable unittest_bit_alloc for now

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #9219 from renhwztetecs/renhw-wip-logmonitor-cleanup

mon/log: clean unused function and comments

Reviewed-by: Sage Weil <sage@redhat.com>

test: file change affected by auth file header optimization.

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

AUTH: optimize header file dependency.
same work as PR: https://github.com/ceph/ceph/pull/9193, https://github.com/ceph/ceph/pull/9161

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

mon/log: using standard macros instead of -1

using standard macros instead of -1

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

mon/log: clean unused function and comments

clean unused function and comments

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #8656 from kmroz/wip-rgw-handler-cleanup-1

rgw-rest: remove redundant virtual functions
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8886 from cbodley/wip-rgw-zone-short-ids

rgw: check for short_zone_id collisions on period update
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9148 from rzarzynski/wip-rgw-15902

rgw: camelcase names of custom attributes in Swift's responses.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9042 from rzarzynski/wip-rgw-15812

rgw: handle errors properly during GET on Swift's DLO.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9149 from cbodley/wip-rgw-zone-modify-master

rgw: add_zone only clears master_zone if --master=false
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9154 from yehudasa/wip-15886

rgw: keep track of written_objs correctly
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9190 from dillaman/wip-15928

librbd: metadata retrieval added to open image state machine

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9191 from dillaman/wip-15930

rbd-mirror: disable librbd caching for replicated images

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9173 from dillaman/wip-15909

rbd-mirror: potential crash during image status update

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #8983 from dzafman/wip-li-rados

tools/rados: Improve list-inconsistent json format

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: misc fix for test_multi.p

remove redundent meta_checkpoint, cause realm.create_user(user) will
call meta_checkpoint by default(via wait_meta = True).

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

OSD: optimize header file dependency to decreace compile time when modify any file
    OSD's left header dependency optimize.
Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

PG: optimize header file dependency to decreace compile time when modify any file
PG Layer's header dependency optimize.
Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

PGBackend: optimize header file dependency to decreace compile time when modify any file

hmm, everytime modify any file, then compile, then wait&wait&wait, so trying to optimize
this part, wish to decrease header dependency, try to improve compile speed.

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

tool: change affected file include relation by mon file header optimize

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

MON: optimize header file dependency.

same work as PR: https://github.com/ceph/ceph/pull/9161

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

crush: fix the problem that can not find the define item below the bucket

Fixes: http://tracker.ceph.com/issues/15154
Signed-off-by: song baisen <song.baisen@zte.com.cn>

Merge pull request #9108 from tchaikov/wip-13990

osd: reset session->osdmap if waiting_for_pg is empty

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8908 from jcsp/wip-15705

mds: fix mdsmap print_summary with standby replays

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tests: disable unittest_bit_alloc for now

See http://tracker.ceph.com/issues/15941

Signed-off-by: Sage Weil <sage@redhat.com>

ObjectCacher: pass new journal tid when overwriting extent

librbd needs to ensure the new journal event has been safely
committed to disk before it can mark an overwritten journal
event as committed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: record rbd CLI debug messages during mirror stress

The debug messages from 'rbd bench-write' and 'rbd snap create',
in addition to the existing debug messages from rbd-mirror, make
it possible to determine the source of any image inconsistency.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9199 from jmunhoz/mc-brp-aws4

doc: rgw: minor cleanup and doc update

Merge pull request #9133 from tchaikov/wip-cmake-selinux

cmake: add selinux support

Reviewed-by: Casey Bodley <cbodley@redhat.com>

journal: reset watch step after pruning expired tag

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: additional debug messages during image replayer start/stop

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: ensure proper handling of status updates during shutdown

Previously, several shutdown race conditions could occur due to the
use of the async work queue for scheduling updates.

Fixes: http://tracker.ceph.com/issues/15909
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: track bootstrap state within image status

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: combine ImageReplayer stopped and uninitialized states

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: lock ordering issue in status update callback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9192 from tchaikov/cmake-object-library

cmake: use object library instead recompiling popular .cc files

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rbd-mirror: disable librbd caching for replicated images

Each image has its own cache and each cache uses its own thread. With
a large replicated cluster, this could result in thousands of extra
threads and gigabytes of extra memory.

Fixes: http://tracker.ceph.com/issues/15930
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9146 from cbodley/wip-rgw-period-push

radosgw-admin: fix 'period push' handling of --url
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: doc: bucket request payment update

S3 doc update.

The bucket request payment feature is available in Jewel

commits:

899d7754fe15ab8635fb506fcadafa8e5e92c376
461bd7fa08710be994c522c4d173a4151dbc173a

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

test: initial python APIs to support mirroring

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd: initial python APIs to support mirroring

Fixes: http://tracker.ceph.com/issues/15656
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rgw: aws4: fix log message

With the UNSIGNED-PAYLOAD bugfix in place this log message requires an update.

Requests asking for unsigned payloads don't need any completion handler so
notifying about body detection in this point is not accurate.

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

Merge pull request #8553 from yatinkumbhare/doc-fix

doc: fix broken link for bootstrapping monitors

cmake: add cython_rados to vstart target

"ceph" cli imports rados python module

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9182 from objoo/YD-mailmap-updates-MAY

Leftover of previous mailmap not merged.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #9143 from tchaikov/wip-osdmaptool-man

man: osdmaptool: document test-map-pgs* options

Reviewed-by: Vikhyat Umrao <vumrao@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

cmake: make common/TextTable.cc an object library

so we don't recompile it whenever it is used.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: reuse secret.c's object library

so we don't need to recompile it when using it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9153 from ceph/openstack_wusui

OpenStack setup scripts.

librbd: metadata retrieval added to open image state machine

Fixes: http://tracker.ceph.com/issues/15928
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: async version of metadata_list helper method

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8669 from dreamhost/wip-15532

rgw/s3website: ErrorDoc & Double-fault support

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #9150 from stiopaa1/compatset_removeIncludeVector_mc

common: CompatSet.h: remove unneeded #include<vector>

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9158 from cxwshawn/wip-tc-fix

osd: erase duplicated header include

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9169 from stiopaa1/osd_hitSetRemoveFormatterInclude

osd/HitSet: remove unneeded #include - Formatter

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9186 from stiopaa1/osd_fixTypoInComment

osd/OSD.h: fix typo in comment

osd/OSD.h: fix typo in comment

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

mailmap: Luo Runbing affiliation (was "runsisi", name normalization between Huazhong University of Science and Technology part & ZTE .)

Signed-off-by: Yann Dupont <yann@objoo.org>

xio: rename enqueue_for_send() to enqueue()Reviewed-by: Sage Weil <sage@redhat.com>

xio: rename enqueue_for_send() to enqueue()

Reviewed-by: Sage Weil <sage@redhat.com>

mailmap: Jenkis affiliation (name normalization)

Signed-off-by: Yann Dupont <yann@objoo.org>

Merge pull request #9181 from dillaman/wip-rbd-mirror-test

qa/workunits/rbd: fixed rbd_mirror teuthology runtime errors

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

mailmap: Fangchen Sun affiliation (and name normalisation, was "sunspot")

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Jacek J. Łakis affiliation (name normalisation)

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Sangdi Xu affiliation (name normalisation)

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Roland Mechler  affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>