rgw_file: add lock protection for readdir against gc

The state.mtx protects state.events accessed both from readdir and gc.
Fixes: http://tracker.ceph.com/issues/20121
Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>

Merge pull request #15275 from ktdreyer/wip-17396

docs: document "osd recovery max single start" setting

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15276 from liewegas/wip-ext4-again

qa/suites/rados/singleton-nomsgr: fix syntax

Merge pull request #15252 from liewegas/wip-cleanup-tell

qa/tasks/ceph_manager: 'ceph $service tell ...' is obsolete

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15250 from neha-ojha/wip-19989

osd/OSDMap.cc: check if osd is out in subtree_type_is_down

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14944 from yuyuyu101/wip-msgr-throttle-ops

ceph_osd: remove client message cap limit

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14655 from chardan/jfw-wip-halflife_atomic_t-gojira

test: migrate atomic_t to std::atomic

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11966 from aclamk/wip-linear-zeros-crc

common: Improved CRC calculation for zero buffers

qa/suites/rados/singleton-nomsgr/health...: whitelist wrongly marked me down

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton-nomsgr: fix syntax

This parsed out as

  tasks:
  - install: null
  - ceph:
      conf:
        osd: osd max object name len = 400 osd max object namespace len = 64
  - workunit:
      clients:
        all:
        - rados/test_health_warnings.sh

which is clearly not correct.

Signed-off-by: Sage Weil <sage@redhat.com>

docs: document "osd recovery max single start" setting

Fixes: http://tracker.ceph.com/issues/17396
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #15028 from pritha-srivastava/wip-rgw-bucket-policy-test

rgw: Adding code to create tenanted user for s3 bucket policy tests.

Merge pull request #15070 from ceph/wip-turn-off-fcgi-tserlin

rgw: Turn off fcgi as a frontend

Merge pull request #15175 from cloudbau/update-rgw-docs

doc: Update some RGW documentation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15213 from jcsp/wip-20040

mon: fix accesing pending_fsmap from peon

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #15130 from ukernel/wip-19946

mds: fix CDir::merge() for mds_debug_auth_pins

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15105 from ukernel/wip-19892

qa/cephfs: disable mds_bal_frag for TestStrays.test_purge_queue_op_rate

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15122 from batrick/test-fragment-error

qa: fix float parse error in test_fragment

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15106 from ukernel/wip-export-pin-cleanup

ceph: simplify CInode::maybe_export_pin()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15129 from ukernel/wip-mds-standby-for-rank

mon/MDSMonitor: respect mds_standby_for_rank config

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15131 from ukernel/wip-19955

mds: limit client writable range increment

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15198 from renhwztetecs/renhw-wip-client-inode

client/inode: fix the dump type of Inode::dump()

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #15264 from xiexingguo/wip-kill-typo

os/bluestore/KernelDevice: fix comments

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15266 from ztczll/master

test:Check make_writeable() return value

test/:Check make_writeable return value

Signed-off-by: zhanglei <243290414@qq.com>

os/bluestore/KernelDevice: fix comments

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15206 from ifed01/wip-fix-rbd-bench

rbd/bench: fix write gaps when doing sequential writes with io-threads > 1

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15238 from xiexingguo/wip-fix-blue-mkfs

os/bluestore: write "mkfs_done" into disk only if we pass fsck() tests

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15228 from chardan/jfw-wip-halflife_atomic_t-filestore

filestore: migrate atomic_t to std::atomic<>

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: 'ceph $service tell ...' is obsolete

This died forever ago; no need for the fallback here.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9307 from cxwshawn/wip-tc7-fix

crush: optimize header file dependency

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14035 from liewegas/wip-bluestore-kv-finisher

os/bluestore: separate kv_sync_thread into two parts

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #14176 from LiumxNL/wip-170321

mon: show io status quickly if no update in a long period

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15169 from joscollin/wip-cleanup-redundant-headers-2

common: Remove redundant includes - 2

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #15177 from liewegas/wip-bluestore-nonexperimental

os: remove experimental status for BlueStore

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15244 from liewegas/wip-crush-features

crush/CrushWrapper: fix has_incompat_choose_args()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15245 from liewegas/wip-20058

qa/suites/rados/*/at-end: wait for healthy before scrubbing

Merge pull request #15190 from tchaikov/wip-ceph-term-width

ceph.in: adjust usage width according to user's tty

Reviewed-by: Dan Mick <dmick@redhat.com>

osd/OSDMap.cc: check if osd is out in subtree_type_is_down

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #15246 from linuxbox2/wip-unlink-safe

rgw_file: release rgw_fh lock and ref on ENOTEMPTY

Merge pull request #15202 from liewegas/wip-19935

qa/suites/rados: temporarily remove scrub_test from basic/ until post-luminous

Merge pull request #15207 from liewegas/wip-20043

qa/suites/rados/singleton-nomsg/health-warnings: behave on ext4

Merge pull request #15092 from yuyuyu101/wip-msgr-writelock

msg/async: reduce write_lock contention

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15172 from badone/wip-erasure-code-optimize-header-file-dependency

erasure-code: optimize header file dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15187 from badone/wip-compressor-optimize-header-file-dependency

compressor: optimize header file dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15201 from liewegas/wip-19990

qa/workunits/rados/test_health_warning: misc fixes

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #15098 from cbodley/wip-rgw-default-frontend

rgw: remove fastcgi from default rgw frontends

Reviewed-by: Sage Weil <sage@redhat.com>

rgw_file: release rgw_fh lock and ref on ENOTEMPTY

An early return in new unlink bucket num_entries check was
missing a conditional unlock and unref.

Fixes: http://tracker.ceph.com/issues/20061
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #15216 from linuxbox2/wip-rgw-file-unlink

rgw_file:  remove post-unlink lookup check

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15179 from liewegas/wip-fix-osd-health

mon: fix a few bugs with the osd health reporting

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #15188 from ShiqiCooperation/master

mon: delete useless function definition

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15204 from joscollin/wip-cleanup-redundant-headers-3

common: Remove redundant includes - 3

qa/suites/rados/*/at-end: wait for healthy before scrubbing

The scrub_pgs command also waits for healthy for a while, but fails
silently if it times out, which means the subsequent scrubs will also
fail to clean up.

This forces an earlier failure that does not obscure the root cause.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14389 from mingshuaiwang/master

rgw: segment fault when shard id out of range

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12774 from shashalu/gc-list

rgw: fix 'gc list --include-all' command infinite loop the first items

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crush/CrushWrapper: fix has_incompat_choose_args()

Special case empty and return false.  Otherwise the rest of the method
will dereference an invalid iterator on an empty choose_args
map.

Fixes broken fix in b1e4295570b6af4d844cf3ec55d77f5c287b29e6.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15242 from liupan1111/wip-fix-msg

msg/async: Lower down the AsyncMessenger's standby warning from debug

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15218 from liewegas/wip-crush-features

crush/CrushWrapper: fix has_incompat_choose_args

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

msg/async: Lower down the AsyncMessenger's standby warning from debug
level 0 to 10.

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

os/bluestore: save "mkfs_done" only if we pass fsck() tests

In our local test bed, we found mkfs() sometimes can fail
due to errors discovered by fsck(), and is therefore unrecoverable
by redoing mkfs() as the "mkfs_done" flag has been successfully saved into disk.

This patch fixes the above case.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #14610 from jcsp/wip-17980

#17980: MDS client blacklisting and blacklist on eviction

Reviewed-by: Yan, Zheng <zyan@redhat.com>

rbd/bench: fixes write gaps when doing sequential writes with io-threads > 1

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>
Signed-off-by: Mykola Golub mgolub@mirantis.com

Merge pull request #14550 from ukernel/wip-multimds-misc

mds: miscellaneous multimds fixes

Reviewed-by: John Spray <john.spray@redhat.com>

qa: refine assert_session_count (don't count killing)

Signed-off-by: John Spray <john.spray@redhat.com>

qa: update TestVolumeClient for new blacklisting

Blacklisted clients will now proactively fail
outstanding operations, rather than blocking.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: clean up whitespace in test_misc.py

Signed-off-by: John Spray <john.spray@redhat.com>

qa: remove outdated TODO in TestVolumeClient

Signed-off-by: John Spray <john.spray@redhat.com>

qa: log whitelist for eviction messages

Signed-off-by: John Spray <john.spray@redhat.com>

mds: improve eviction usability and update docs

The "session ls" and "session evict" are now
"client ls" and "client evict" (the old ones
are still there for backwards compatibility).

The automatic client eviction now emits
cluster logs that call the client by its
friendly name (usually the hostname).

Signed-off-by: John Spray <john.spray@redhat.com>

mds: tidy up 'kill_session' naming

We had two kill_sessions.  Leave Server::kill_session (the
one that actually operates on MDS sessions only), and rename
the MDSRank one to evict_client, because it does more than
just kill the MDS session.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: add test_filelock_eviction

To check that eviction is releasing flocks.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: hook up autoclose to blacklisting eviction

Signed-off-by: John Spray <john.spray@redhat.com>

mds: make blacklisting configurable in kill_session

This makes the code a bit awkward because we now have
4x paths through this function (blocking, noblocking, blacklisting,
nonblacklisting).

Signed-off-by: John Spray <john.spray@redhat.com>

client: fix unmounting while blacklisted

Previously, if you ever made it as far as ~Client
with any inodes in your cache, you'd hit the ObjectCacher
is_locked_by_me assertion during release_set on those inode's osets.

Signed-off-by: John Spray <john.spray@redhat.com>

client: detect blacklisting of myself

EIO all operations when this happens, and do a cut-down
unmount() so that once the user has noticed his mount
is broken, he can unmount it without getting stuck
waiting for flushes/caps/sessions.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: update osd epoch barrier when handling blacklist

Signed-off-by: John Spray <john.spray@redhat.com>

mds: update osdmap epoch barrier on eviction

Signed-off-by: John Spray <john.spray@redhat.com>

mds: blacklist clients on eviction

Fixes: http://tracker.ceph.com/issues/9754
Signed-off-by: John Spray <john.spray@redhat.com>

mon: permit MDS daemons to do blacklisting

Signed-off-by: John Spray <john.spray@redhat.com>

mds: remove stale comment

The code the comment is talking about went
away in 2490a0a1e.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: apply OSDMap blacklist to CephFS clients

Signed-off-by: John Spray <john.spray@redhat.com>

osd/OSDMap: std::set version of get_blacklist

Signed-off-by: John Spray <john.spray@redhat.com>

osdc: expose new blacklist entries from Objecter

For the MDS which needs to know about new blacklist entries
in order to evict corresponding MDS sessions.

Fixes: http://tracker.ceph.com/issues/17980
Signed-off-by: John Spray <john.spray@redhat.com>

filestore: migrate atomic_t to std::atomic<>

Signed-off-by: Jesse Williamson <jwilliamson@suse.de>

Merge pull request #15220 from joscollin/wip-cleanup-rados-typo

include/rados: Fix typo in rados_ioctx_cct() doc

include/rados: Fix typo in rados_ioctx_cct() doc

This fixes PR#15077.

Signed-off-by: Jos Collin <jcollin@redhat.com>

ceph.in: adjust usage width according to user's tty

fixed a pep8 warning also

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/ceph_daemon.py: move _gettermsize() into Termsize

as the latter is the only consumer of _gettermsize(). and a little bit
refactor to improve the readability and be more pep8 compliant.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crush/CrushWrapper: fix has_incompat_choose_args

If the map has *no* choose_args it does not have incompat choose args.
This makes the OSDMap::get_features() check

  if (crush->has_incompat_choose_args())
    features |= CEPH_FEATURE_CRUSH_CHOOSE_ARGS;

happier.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw_file:  remove post-unlink lookup check

This could induce asserts in multi-nfs-gateway scenarios.

Fixes: http://tracker.ceph.com/issues/20047
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #15210 from linuxbox2/wip-rgw-rmbuck

rgw_file: remove hidden uxattr objects from buckets on delete

rgw_file: remove hidden uxattr objects from buckets on delete

If a setattr (e.g., chown) has been performed on a bucket, then like
any directory it has a hidden object storing its attributes. This must
be deleted before attempting bucket delete, otherwise, actually empty
buckets will not be removable via NFS.

Fixes: http://tracker.ceph.com/issues/20045
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

mon: fix accesing pending_fsmap from peon

Fixes: http://tracker.ceph.com/issues/20040
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15195 from theanalyst/doc/12.0.3-typo-fix

doc: fixing an error in 12.0.3 release notes

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

common: Remove redundant includes

Fixes: http://tracker.ceph.com/issues/19883 (Partially)
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15186 from adamemerson/wip-clangtastic-mr-clocks

rgw: Remove pessimizing move
common: Forward-declare container I/O overloads

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/singleton-nomsg/health-warnings: behave on ext4

We may land on an ext4 root partition.

Fixes: http://tracker.ceph.com/issues/20043
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12861 from theanalyst/wip-absolute-urls

rgw: pass authentication domain to civetweb

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15150 from ovh/bp-doc-perf-histograms

doc: document perf historgrams

Reviewed-by: Sage Weil <sage@redhat.com>

doc: document perf historgrams

Documentation for perf histograms schema and dump.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>