qa/suites/rados/*/at-end: wait for healthy before scrubbing

The scrub_pgs command also waits for healthy for a while, but fails
silently if it times out, which means the subsequent scrubs will also
fail to clean up.

This forces an earlier failure that does not obscure the root cause.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14389 from mingshuaiwang/master

rgw: segment fault when shard id out of range

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12774 from shashalu/gc-list

rgw: fix 'gc list --include-all' command infinite loop the first items

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15242 from liupan1111/wip-fix-msg

msg/async: Lower down the AsyncMessenger's standby warning from debug

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15218 from liewegas/wip-crush-features

crush/CrushWrapper: fix has_incompat_choose_args

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

msg/async: Lower down the AsyncMessenger's standby warning from debug
level 0 to 10.

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

Merge pull request #14610 from jcsp/wip-17980

#17980: MDS client blacklisting and blacklist on eviction

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #14550 from ukernel/wip-multimds-misc

mds: miscellaneous multimds fixes

Reviewed-by: John Spray <john.spray@redhat.com>

qa: refine assert_session_count (don't count killing)

Signed-off-by: John Spray <john.spray@redhat.com>

qa: update TestVolumeClient for new blacklisting

Blacklisted clients will now proactively fail
outstanding operations, rather than blocking.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: clean up whitespace in test_misc.py

Signed-off-by: John Spray <john.spray@redhat.com>

qa: remove outdated TODO in TestVolumeClient

Signed-off-by: John Spray <john.spray@redhat.com>

qa: log whitelist for eviction messages

Signed-off-by: John Spray <john.spray@redhat.com>

mds: improve eviction usability and update docs

The "session ls" and "session evict" are now
"client ls" and "client evict" (the old ones
are still there for backwards compatibility).

The automatic client eviction now emits
cluster logs that call the client by its
friendly name (usually the hostname).

Signed-off-by: John Spray <john.spray@redhat.com>

mds: tidy up 'kill_session' naming

We had two kill_sessions.  Leave Server::kill_session (the
one that actually operates on MDS sessions only), and rename
the MDSRank one to evict_client, because it does more than
just kill the MDS session.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: add test_filelock_eviction

To check that eviction is releasing flocks.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: hook up autoclose to blacklisting eviction

Signed-off-by: John Spray <john.spray@redhat.com>

mds: make blacklisting configurable in kill_session

This makes the code a bit awkward because we now have
4x paths through this function (blocking, noblocking, blacklisting,
nonblacklisting).

Signed-off-by: John Spray <john.spray@redhat.com>

client: fix unmounting while blacklisted

Previously, if you ever made it as far as ~Client
with any inodes in your cache, you'd hit the ObjectCacher
is_locked_by_me assertion during release_set on those inode's osets.

Signed-off-by: John Spray <john.spray@redhat.com>

client: detect blacklisting of myself

EIO all operations when this happens, and do a cut-down
unmount() so that once the user has noticed his mount
is broken, he can unmount it without getting stuck
waiting for flushes/caps/sessions.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: update osd epoch barrier when handling blacklist

Signed-off-by: John Spray <john.spray@redhat.com>

mds: update osdmap epoch barrier on eviction

Signed-off-by: John Spray <john.spray@redhat.com>

mds: blacklist clients on eviction

Fixes: http://tracker.ceph.com/issues/9754
Signed-off-by: John Spray <john.spray@redhat.com>

mon: permit MDS daemons to do blacklisting

Signed-off-by: John Spray <john.spray@redhat.com>

mds: remove stale comment

The code the comment is talking about went
away in 2490a0a1e.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: apply OSDMap blacklist to CephFS clients

Signed-off-by: John Spray <john.spray@redhat.com>

osd/OSDMap: std::set version of get_blacklist

Signed-off-by: John Spray <john.spray@redhat.com>

osdc: expose new blacklist entries from Objecter

For the MDS which needs to know about new blacklist entries
in order to evict corresponding MDS sessions.

Fixes: http://tracker.ceph.com/issues/17980
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15220 from joscollin/wip-cleanup-rados-typo

include/rados: Fix typo in rados_ioctx_cct() doc

include/rados: Fix typo in rados_ioctx_cct() doc

This fixes PR#15077.

Signed-off-by: Jos Collin <jcollin@redhat.com>

crush/CrushWrapper: fix has_incompat_choose_args

If the map has *no* choose_args it does not have incompat choose args.
This makes the OSDMap::get_features() check

  if (crush->has_incompat_choose_args())
    features |= CEPH_FEATURE_CRUSH_CHOOSE_ARGS;

happier.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15210 from linuxbox2/wip-rgw-rmbuck

rgw_file: remove hidden uxattr objects from buckets on delete

rgw_file: remove hidden uxattr objects from buckets on delete

If a setattr (e.g., chown) has been performed on a bucket, then like
any directory it has a hidden object storing its attributes. This must
be deleted before attempting bucket delete, otherwise, actually empty
buckets will not be removable via NFS.

Fixes: http://tracker.ceph.com/issues/20045
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #15195 from theanalyst/doc/12.0.3-typo-fix

doc: fixing an error in 12.0.3 release notes

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #15186 from adamemerson/wip-clangtastic-mr-clocks

rgw: Remove pessimizing move
common: Forward-declare container I/O overloads

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12861 from theanalyst/wip-absolute-urls

rgw: pass authentication domain to civetweb

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15150 from ovh/bp-doc-perf-histograms

doc: document perf historgrams

Reviewed-by: Sage Weil <sage@redhat.com>

doc: document perf historgrams

Documentation for perf histograms schema and dump.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

rgw: Remove pessimizing move

Do not std::move values from returned functions.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Forward-declare container I/O overloads

This will allow strict ADL to find the right overload in the case of a
container of containers and unbreak compilation with Clang.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #15162 from dillaman/wip-19970

librbd: reduce potential of erroneous blacklisting on image close

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

doc: fixing an error in 12.0.3 release notes

Thanks @smithfarm for pointing this out

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #15191 from Linbing/modify-note-word-error

mon: Fixed typo in @post of _active()

Note word error of _active(), take iif with if

Signed-off-by: Linbing <hawkerous@gmail.com>

Merge pull request #15181 from tchaikov/wip-cxx1z-size

common: add ceph::size()

Reviewed-By: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #14946 from dmick/wip-mgr-dashboard

mgr: dashboard GUI module

Merge pull request #14994 from theanalyst/f/tier-fix

rgw_rados: create sync module instances only if run_sync_thread is set

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test: add mgr/ and smoke test for mgr dashboard

Signed-off-by: Dan Mick <dan.mick@redhat.com>

rgw_rados: create sync module instances only if run_sync_thread is set

This allows for radosgw-admin to run even if there was a
misconfiguration in setting tier types which would otherwise make it
difficult to revert an invalid tier type setting. Since sync modules are
necessary only for rgw and not for the admin itself, this allows for
invalid tier configurations to be corrected as rgw-admin wouldn't fail

Fixes: http://tracker.ceph.com/issues/19830
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #15153 from zhangsw/optimize-torrent

rgw: optimize generating torrent file. Object data won't stay in memory  now.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #15001 from chardan/jfw-wip-halflife_atomic_t-kamakuras

rgw: migrate atomic_t to std::atomic<>

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14827 from liuyuhong66/rgw-swift-auth-improve

rgw/rgw_swift_auth.cc: using string::back() instead as the C++11 recommend

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15025 from mikulely/jiaying-fix-lc-list

rgw: fix lc list failure when shards not be all created

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12300 from idealguo/rename-s3_code

rgw: rename s3_code to err_code for swift

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15045 from shashalu/fix-slo

rgw: fix X-Object-Meta-Static-Large-Object in SLO download

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14800 from theanalyst/doc/release-notes/sse-c-correction

doc: release-notes clarify about rgw encryption

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14922 from ukernel/wip-kcephfs-whitelist

qa: update log whitelists for kcephfs suite

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14941 from wjwithagen/wip-wjw-freebsd-doc-manual-install

doc: add FreeBSD manual install

Merge pull request #14988 from ShiqiCooperation/master

test/unittest_bluefs: check whether mounted success

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14688 from cbodley/wip-rgw-multi-suite

qa/rgw: add multisite suite to configure and run multisite tests

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15071 from cbodley/wip-qa-dnsmasq

qa: add task for dnsmasq configuration

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #15090 from theanalyst/release/v12.0.3/notes

doc: release notes for v12.0.3 (dev)

Merge pull request #15133 from cbodley/wip-doc-rgw-remove-fcgi

doc/rgw: remove fastcgi page and sample configs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15144 from drunkard/master

doc: typo fixes on hyperlink/words

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15161 from dzafman/wip-test-fix

test: Test fix for SnapSet change

Reviewed-by: Sage Weil <sage@redhat.com>

common: add ceph::size()

backport from C++17

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15147 from bassam/pr-speedup-ccache

cmake: improved build speed by 5x when using ccache

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15158 from liewegas/wip-mgr-dump

mon/MgrMonitor: add 'mgr dump [epoch]' command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15159 from LiumxNL/fix-ping-monitor

pybind/rados: avoid call free() on invalid pointer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15164 from tchaikov/wip-check-gperf-before-using-it

cmake: check the existence of gperf before using it

Reviewed-By: Adam C. Emerson <aemerson@redhat.com>

librbd: block RPC requests before unregistering watch during close

The watch should be released after exclusive lock is released to
avoid a potential race condition which would result in the
client being blacklisted by a peer.

Fixes: http://tracker.ceph.com/issues/19970
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15089 from zhsj/add-mirror

doc: add new cn ceph mirror to doc and mirroring

Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15173 from joscollin/wip-test-crush-warning

test/crush: silence warnings from -Walloc-size-larger-than= and -Wstringop-overflow=

Merge pull request #15171 from ztczll/master

test/objectstore: Check apply_transaction() return values

Merge pull request #15034 from liewegas/wip-bluestore-sync-write

os/bluestore/KernelDevice: fix sync write vs flush

Reviewed-by: Varada Kari <varada.kari@gmail.com>

os/bluestore/KernelDevice: fix flush() vs sync write()

We need to set the atomic flag so that a flush will actually flush.

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: introduced support for blocking watch notifications

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: recheck current lock owner before blacklist

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/crush: silence warnings from -Walloc-size-larger-than= and -Wstringop-overflow=

The following warnings appear during make:

/home/jcollin/workspace/ceph/src/test/crush/CrushWrapper.cc: In member function ‘virtual void CrushWrapper_choose_args_compat_Test::TestBody()’:
/home/jcollin/workspace/ceph/src/test/crush/CrushWrapper.cc:1057:20: warning: argument 1 value ‘18446744073709550912’ exceeds maximum object size 9223372036854775807 [-Walloc-size-larger-than=]
   crush_choose_arg choose_args[c.get_max_buckets()];
                    ^~~~~~~~~~~

In function ‘void* memset(void*, int, size_t)’,
    inlined from ‘virtual void CrushWrapper_choose_args_compat_Test::TestBody()’ at /home/jcollin/workspace/ceph/src/test/crush/CrushWrapper.cc:1058:9:
/usr/include/bits/string3.h:90:70: warning: ‘void* __builtin_memset(void*, int, long unsigned int)’: specified size 18446744073709550912 exceeds maximum object size 9223372036854775807 [-Wstringop-overflow=]
   return __builtin___memset_chk (__dest, __ch, __len, __bos0 (__dest));

Signed-off-by: Jos Collin <jcollin@redhat.com>

Check apply_transaction return values

Signed-off-by: zhanglei <243290414@qq.com>

doc: v12.0.3 release notes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: PendingReleaseNotes move the 12.0.2 section of release notes

So that we can track whatever diff goes in after

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: segment fault when shard id out of range

Fixes: http://tracker.ceph.com/issues/19732
Signed-off-by: redickwang <redickwang@tencent.com>

ceph.spec.in, debian/control: build/runtime dependency for cherrypy

Signed-off-by: Dan Mick <dan.mick@redhat.com>

mgr: change config_prefix to be 'mgr/<id>/' and separate handle/key by '/'

dm_crypt config-key usage has established '/' as the separator; keep
that as a convention for now

Signed-off-by: Dan Mick <dan.mick@redhat.com>

mgr/PyModules: move constructor definition to .cc

Otherwise, the incomplete ServeThread type causes problems
for others who include PyModules.h

Signed-off-by: Dan Mick <dan.mick@redhat.com>

pybind/mgr/dashboard/module.py: allow custom server addr, port

Signed-off-by: Dan Mick <dan.mick@redhat.com>

mgr/PyModules.cc,PyState.cc: improve config access debug logging

Signed-off-by: Dan Mick <dan.mick@redhat.com>

client/SyntheticClient.cc: filer is now a std::unique_ptr

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ceph_syn.cc, client/SyntheticClient.cc: Client -> StandaloneClient

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ServeThread: add 'running' flag, don't notify non-running threads

Signed-off-by: Dan Mick <dan.mick@redhat.com>

pybind/mgr: web dashboard

A read only display of some of the cluster's state, plus
some cephfs/rbd/osd pages that demonstrate how to go
read state remotely from daemons.

Signed-off-by: John Spray <john.spray@redhat.com>

vstart: enable gui plugin

Signed-off-by: John Spray <john.spray@redhat.com>

osd: wire up "histogram dump" to `tell`

...so that mgr modules can get at it.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15148 from zealoussnow/wip-spec

build: move bash_completion.d/ceph to ceph-common

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: check the existence of gperf before using it

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15160 from bassam/pr-system-boost

cmake: fix boost components for WITH_SYSTEM_BOOST

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15107 from guihecheng/rgw_file-cleanup

rgw: use pre-defined calls to replace raw flag operation

doc: hyperlink fixes

Signed-off-by: Drunkard Zhang <gongfan193@gmail.com>

doc: typo fixes

Signed-off-by: Drunkard Zhang <gongfan193@gmail.com>