crush: update choose_args on adjust_item_weight

The osd_crush_update_weight_set (true by default) can be used to disable
the update of the weights.

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: update choose_args when items are added/removed

When items are added:

- the value in the weight set is set to the target weight. It is assumed
  that weight set are updated on a regular basis and will eventually be
  set to a value that prevents excessive over/under filling.

- the value in the id list is set to the item id.

When items are removed, their weight / ids in all choose_args are
removed.

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: allow item addition/removal when choose_args exists

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: wrap crush_bucket_{remove,add}_item

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #14658 from chardan/jfw-wip-halflife_atomic_t-baragon

librados,libradosstriper,test: migrate atomic_t to std::atomic (baragon)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15227 from tchaikov/wip-20051

mon/MonClient: cancel pending commands on shutdown

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15249 from Alessandro-Barbieri/patch-2

yasm-wrapper: filter -pthread

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15268 from ifed01/wip-bluestore-false-assert

os/bluestore: fix false assert in IOContext::aio_wake

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15272 from tchaikov/wip-mon-cleanups

mon: cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15192 from badone/wip-tracker-19900-wait_auth_rotating-crash

mgr: Mark session connections down on shutdown

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15146 from tchaikov/wip-split-test-tiering

qa: split test_tiering into smaller pieces

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15156 from tchaikov/wip-ceph.in-assert-connected

ceph.in: assert(state==connected) before help_for_target()

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15224 from tchaikov/wip-denc-no-deep-copy-for-segmented-buffer

denc: add need_contiguous to denc_traits

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15062 from ukernel/wip-19912

qa/tasks/cephfs: use getattr to guarantee inode is in client cache

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15270 from ceph/wip-subman-cron-attr-tserlin

Set subman cron attributes in spec file

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #15253 from tchaikov/wip-ceph-daemon-termsize

pybind/ceph_daemon.py: fix Termsize.update

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #15279 from joscollin/wip-cleanup-rename-filejournal-object

test: Rename FileJournal object to distinguish

Set subman cron attributes in spec file

Fixes: http://tracker.ceph.com/issues/20074
Signed-off-by: Thomas Serlin <tserlin@redhat.com>

Merge pull request #15200 from smithfarm/wip-rdma-devel

build/ops: rpm: move RDMA and python-prettytables build dependencies to distro-conditional section

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

test: Rename FileJournal object to distinguish

Renamed FileJournal object for easier searching and identifying with in test_filejournal.cc

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15258 from big-henry/wip-mds-misc-fix

mds: fix client ID truncation

Merge pull request #15235 from smithfarm/wip-doc-empowers

doc: fix factual inaccuracy in doc/architecture.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15275 from ktdreyer/wip-17396

docs: document "osd recovery max single start" setting

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15276 from liewegas/wip-ext4-again

qa/suites/rados/singleton-nomsgr: fix syntax

Merge pull request #15252 from liewegas/wip-cleanup-tell

qa/tasks/ceph_manager: 'ceph $service tell ...' is obsolete

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15250 from neha-ojha/wip-19989

osd/OSDMap.cc: check if osd is out in subtree_type_is_down

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14944 from yuyuyu101/wip-msgr-throttle-ops

ceph_osd: remove client message cap limit

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14655 from chardan/jfw-wip-halflife_atomic_t-gojira

test: migrate atomic_t to std::atomic

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11966 from aclamk/wip-linear-zeros-crc

common: Improved CRC calculation for zero buffers

qa/suites/rados/singleton-nomsgr/health...: whitelist wrongly marked me down

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton-nomsgr: fix syntax

This parsed out as

  tasks:
  - install: null
  - ceph:
      conf:
        osd: osd max object name len = 400 osd max object namespace len = 64
  - workunit:
      clients:
        all:
        - rados/test_health_warnings.sh

which is clearly not correct.

Signed-off-by: Sage Weil <sage@redhat.com>

docs: document "osd recovery max single start" setting

Fixes: http://tracker.ceph.com/issues/17396
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #15028 from pritha-srivastava/wip-rgw-bucket-policy-test

rgw: Adding code to create tenanted user for s3 bucket policy tests.

Merge pull request #15070 from ceph/wip-turn-off-fcgi-tserlin

rgw: Turn off fcgi as a frontend

mon/PGMonitor: assert(format_version == 1)

we have been PGMonitor's formart_version has been 1 since v0.67
(dumpling). so let's drop it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon,mgr: std::move(osd_stat) when possible

osd_stat_t could be a relatively large struct, so use std::move() when
possible

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15175 from cloudbau/update-rgw-docs

doc: Update some RGW documentation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mon/MonClient: cancel pending commands on shutdown

Fixes: http://tracker.ceph.com/issues/20051
Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: drop unused param

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15213 from jcsp/wip-20040

mon: fix accesing pending_fsmap from peon

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #15130 from ukernel/wip-19946

mds: fix CDir::merge() for mds_debug_auth_pins

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15105 from ukernel/wip-19892

qa/cephfs: disable mds_bal_frag for TestStrays.test_purge_queue_op_rate

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15122 from batrick/test-fragment-error

qa: fix float parse error in test_fragment

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15106 from ukernel/wip-export-pin-cleanup

ceph: simplify CInode::maybe_export_pin()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15129 from ukernel/wip-mds-standby-for-rank

mon/MDSMonitor: respect mds_standby_for_rank config

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15131 from ukernel/wip-19955

mds: limit client writable range increment

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15198 from renhwztetecs/renhw-wip-client-inode

client/inode: fix the dump type of Inode::dump()

Reviewed-by: Yan, Zheng <zyan@redhat.com>

os/bluestore: fix false assertion

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: wrap IOContext::aio_wake with additional stuff to remove copy-paste for its use

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15264 from xiexingguo/wip-kill-typo

os/bluestore/KernelDevice: fix comments

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15266 from ztczll/master

test:Check make_writeable() return value

test/:Check make_writeable return value

Signed-off-by: zhanglei <243290414@qq.com>

os/bluestore/KernelDevice: fix comments

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15206 from ifed01/wip-fix-rbd-bench

rbd/bench: fix write gaps when doing sequential writes with io-threads > 1

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

pybind/ceph_daemon.py: fix Termsize.update

it's a regression introduced by 84f9c12

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: Mark session connections down on shutdown

Also assert in OSD::ms_get_authorizer since it is dangerous to proceed
if we are shutting down.

Fixes: http://tracker.ceph.com/issues/19900
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

denc: add need_contiguous to denc_traits

so the decode() will not try to copy the bufferlist for a continous one
if the bufferlist is segmented *and* its length is greater than
CEPH_PAGE_SIZE (4K) if the decoded object does not "need_contiguous" per
its denc_traits<>.

copying a memory chunk could be expensive if the decoded bufferlist is
huge, so we should try to avoid this. this could happen when we read the
buffer from bluestore.

and drop the partial specialization for denc() which tries to differentiate
traits::featured and !traits::featured, it does not matter to
decode() if the type supports feature or not. the encode() does.

also use denc() in interval_set<>::decode(). unlike encode(), decode(),
encode_nohead() or decode_nohead(), denc() is not part of our legacy
dencoder, so i think it's fine and encouraged to use it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15238 from xiexingguo/wip-fix-blue-mkfs

os/bluestore: write "mkfs_done" into disk only if we pass fsck() tests

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15228 from chardan/jfw-wip-halflife_atomic_t-filestore

filestore: migrate atomic_t to std::atomic<>

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: 'ceph $service tell ...' is obsolete

This died forever ago; no need for the fallback here.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9307 from cxwshawn/wip-tc7-fix

crush: optimize header file dependency

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14035 from liewegas/wip-bluestore-kv-finisher

os/bluestore: separate kv_sync_thread into two parts

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #14176 from LiumxNL/wip-170321

mon: show io status quickly if no update in a long period

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15169 from joscollin/wip-cleanup-redundant-headers-2

common: Remove redundant includes - 2

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #15177 from liewegas/wip-bluestore-nonexperimental

os: remove experimental status for BlueStore

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15244 from liewegas/wip-crush-features

crush/CrushWrapper: fix has_incompat_choose_args()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15245 from liewegas/wip-20058

qa/suites/rados/*/at-end: wait for healthy before scrubbing

Merge pull request #15190 from tchaikov/wip-ceph-term-width

ceph.in: adjust usage width according to user's tty

Reviewed-by: Dan Mick <dmick@redhat.com>

osd/OSDMap.cc: check if osd is out in subtree_type_is_down

Signed-off-by: Neha Ojha <nojha@redhat.com>

filter -pthread

Get confused for a yasm option

Signed-off-by: Alessandro Barbieri <ale.barbio@alice.it>

Merge pull request #15246 from linuxbox2/wip-unlink-safe

rgw_file: release rgw_fh lock and ref on ENOTEMPTY

Merge pull request #15202 from liewegas/wip-19935

qa/suites/rados: temporarily remove scrub_test from basic/ until post-luminous

Merge pull request #15207 from liewegas/wip-20043

qa/suites/rados/singleton-nomsg/health-warnings: behave on ext4

Merge pull request #15092 from yuyuyu101/wip-msgr-writelock

msg/async: reduce write_lock contention

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15172 from badone/wip-erasure-code-optimize-header-file-dependency

erasure-code: optimize header file dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15187 from badone/wip-compressor-optimize-header-file-dependency

compressor: optimize header file dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15201 from liewegas/wip-19990

qa/workunits/rados/test_health_warning: misc fixes

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #15098 from cbodley/wip-rgw-default-frontend

rgw: remove fastcgi from default rgw frontends

Reviewed-by: Sage Weil <sage@redhat.com>

rgw_file: release rgw_fh lock and ref on ENOTEMPTY

An early return in new unlink bucket num_entries check was
missing a conditional unlock and unref.

Fixes: http://tracker.ceph.com/issues/20061
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #15216 from linuxbox2/wip-rgw-file-unlink

rgw_file:  remove post-unlink lookup check

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15179 from liewegas/wip-fix-osd-health

mon: fix a few bugs with the osd health reporting

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #15188 from ShiqiCooperation/master

mon: delete useless function definition

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15204 from joscollin/wip-cleanup-redundant-headers-3

common: Remove redundant includes - 3

qa/suites/rados/*/at-end: wait for healthy before scrubbing

The scrub_pgs command also waits for healthy for a while, but fails
silently if it times out, which means the subsequent scrubs will also
fail to clean up.

This forces an earlier failure that does not obscure the root cause.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14389 from mingshuaiwang/master

rgw: segment fault when shard id out of range

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12774 from shashalu/gc-list

rgw: fix 'gc list --include-all' command infinite loop the first items

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crush/CrushWrapper: fix has_incompat_choose_args()

Special case empty and return false.  Otherwise the rest of the method
will dereference an invalid iterator on an empty choose_args
map.

Fixes broken fix in b1e4295570b6af4d844cf3ec55d77f5c287b29e6.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15242 from liupan1111/wip-fix-msg

msg/async: Lower down the AsyncMessenger's standby warning from debug

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15218 from liewegas/wip-crush-features

crush/CrushWrapper: fix has_incompat_choose_args

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

include/denc: templatize denc() for integer types

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async: Lower down the AsyncMessenger's standby warning from debug
level 0 to 10.

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

doc: doc/architecture.rst: unify initial caps in term Ceph Monitor

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: fix factual inaccuracy in doc/architecture.rst

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

os/bluestore: save "mkfs_done" only if we pass fsck() tests

In our local test bed, we found mkfs() sometimes can fail
due to errors discovered by fsck(), and is therefore unrecoverable
by redoing mkfs() as the "mkfs_done" flag has been successfully saved into disk.

This patch fixes the above case.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #14610 from jcsp/wip-17980

#17980: MDS client blacklisting and blacklist on eviction

Reviewed-by: Yan, Zheng <zyan@redhat.com>

rbd/bench: fixes write gaps when doing sequential writes with io-threads > 1

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>
Signed-off-by: Mykola Golub mgolub@mirantis.com

Merge pull request #14550 from ukernel/wip-multimds-misc

mds: miscellaneous multimds fixes

Reviewed-by: John Spray <john.spray@redhat.com>

qa: refine assert_session_count (don't count killing)

Signed-off-by: John Spray <john.spray@redhat.com>

qa: update TestVolumeClient for new blacklisting

Blacklisted clients will now proactively fail
outstanding operations, rather than blocking.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: clean up whitespace in test_misc.py

Signed-off-by: John Spray <john.spray@redhat.com>