misc: mark functions with 'override' specifier

Fix for:

[src/*/*.h:XX] -> [src/*/*.h:XX]: (style) The function 'XYZ' overrides a
 function in a base class but is not marked with a 'override' specifier.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge PR #21762 into master

* refs/pull/21762/head:
mds: mark new root inode dirty

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21652 into master

* refs/pull/21652/head:
client: use common interp of st_nlink for dirs

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21715 into master

* refs/pull/21715/head:
mds: don't report slow request for blocked filelock request

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21010 into master

* refs/pull/21010/head:
common: refactor for array size
common: Use ARRAY_SIZE macro
common: compress return logic
common: Remove multiple assignment
common: Remove unnecessary semicolon
common: addr_parsing: Cleanup and Refactor Code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21351 from votdev/dashboard_rgw_management

mgr/dashboard: Add RGW user and bucket management features

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #21756 from liewegas/wip-23949

mon/OSDMonitor: set FLAG_SELFMANAGED_SNAPS on cephfs snap removal

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #21757 from liewegas/wip-osdc-warning

osdc/Objecter: fix warning

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: mark new root inode dirty

Fixes: http://tracker.ceph.com/issues/23960
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21696 from jan--f/prometheus-file_sd-consider-defaultport

pybing/mgr/prometheus: return default port if config-key get returns …

Reveiwed-by: John Spray <john.spray@redhat.com>

Merge pull request #21749 from b-ranto/wip-expose-time-perfcount

prometheus: Handle the TIME perf counter type metrics

Reveiwed-by: John Spray <john.spray@redhat.com>

Merge pull request #21627 from jcsp/wip-mgr-dashboard-ssl

mgr/dashboard: add TLS

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

osdc/Objecter: fix warning

Fallout from 891f5192427a4a783d5d7194fc2556dfdc1a0ed2

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: set FLAG_SELFMANAGED_SNAPS on cephfs snap removal

CephFS uses a different path to remove selfmanaged snaps than librados,
so while the librados path goes through pg_pool_t::remove_unmanaged_snap(),
we open code the snap addition to the pool's removed_snaps here.  If we
don't set FLAG_SELFMANAGED_SNAPS at that time, we will implicitly set it
during decode and get a CRC mismatch.

Fix by explicitly setting FLAG_SELFMANAGED_SNAPS flag here.

Fixes: http://tracker.ceph.com/issues/23949
Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: dump osdmaps if crc doesn't match

Dump both the json and hexdump at debug level 20.

Hunting http://tracker.ceph.com/issues/23949

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21742 from liewegas/wip-23940

osdc/Objecter: fix recursive locking in _finish_command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21745 from liewegas/wip-pg-removal-race

osd: fix _process handling for pg vs slot race

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #20678 from ceph/wip-s3a-fix

fix s3atests that are failing for sometime

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20894 from ZVampirEM77/wip-multisite-cleanup

rgw: some cleanup for sync status

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21647 from yehudasa/wip-23859

rgw: fix for issue #21647

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21648 from yehudasa/wip-cloud-sync-7

rgw: cloud sync fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21678 from idiv-biodiversity/wip-doc-scrub_load_threshold

doc: fix error in osd scrub load threshold

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21727 from trociny/wip-23929

librbd: release lock executing deep copy progress callback

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mds: don't report slow request for blocked filelock request

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: http://tracker.ceph.com/issues/22428

Merge PR #21719 into master

* refs/pull/21719/head:
mds: trim log during shutdown to clean metadata

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21720 into master

* refs/pull/21720/head:
mds: kick rdlock if waiting for dirfragtreelock

Reviewed-by: Zheng Yan <zyan@redhat.com>

prometheus: Handle the TIME perf counter type metrics

This patch correctly sets the PERFCOUNTER_MASK to 3 so that the
PERFCOUNTER_TIME metrics are not ignored by the mgr_module code. It also
converts the TIME metrics from nanoseconds to seconds just like the ceph
perf dump does and exposes the metrics via prometheus module.

Signed-off-by: Boris Ranto <branto@redhat.com>

common: refactor for array size

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd: fix _process handling for pg vs slot race

We could see the slot with a different PG than we expected if the old
PG was removed and a new one was instantiated in its place.  We can't
just pick up the new PG pointer, however, since it isn't locked.

Fix by retrying with the slot's new pg (possibly null!).  Move this check
below the other cases so that we know we are otherwise consistent with
the slot, since the next pass around we might get pg==null and skip the
to_process.empty() and requeue_seq checks entirely.

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: release lock executing deep copy progress callback

Fixes: http://tracker.ceph.com/issues/23929
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21706 from liewegas/wip-23860

osd/PG: fix DeferRecovery vs AllReplicasRecovered race

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mds: kick rdlock if waiting for dirfragtreelock

Fixes: https://tracker.ceph.com/issues/23919
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: trim log during shutdown to clean metadata

Otherwise the trimming won't advance so that the remaining inodes are marked
clean.

Fixes: http://tracker.ceph.com/issues/23923
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21702 from theanalyst/wip-std-mutex

osdc/Objector: use std::shared_mutex instead of boost::shared_mutex

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #21731 into master

* refs/pull/21731/head:
client: drop function _get_inodeno

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: silence E741

This is a pretty questionable check because it complains
about the caller of an API instead of the API itself, if
one of the API's members/arguments is one of the
forbidden variable names such as 'O'.

The interface to pyopenssl includes an 'O' member
on the certificate object.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #21743 from yuriw/wip-yuriw-crontab

qa/tests: removed rest suite from the mix

Merge pull request #21697 from dillaman/wip-18753-1

rbd-mirror: additional thrasher testing

Reviewed-by: Mykola Golub <mgolub@suse.com>

qa/tests: removed rest suite from the mix

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #21716 from smithfarm/wip-drop-obs-kludge

build/ops: rpm: Revert "ceph.spec: work around build.opensuse.org"

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>

Merge pull request #21739 from tchaikov/wip-23922

qa/suites/rados/thrash-old-clients: ms_type=simple

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21685 from alfredodeza/wip-rm23874

ceph-volume  failed ceph-osd --mkfs command doesn't halt the OSD creation process

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

osdc/Objecter: fix recursive locking in _finish_command

The path

#9  Objecter::_finish_command (this=this@entry=0x7f76c00aeb30, c=c@entry=0x7f76b0000b10, r=<optimized out>, rs="osd down") at /build/ceph-13.0.2-1932-g458b4fb/src/osdc/Objecter.cc:4950
#10 0x00007f76d26de106 in Objecter::_check_command_map_dne (this=this@entry=0x7f76c00aeb30, c=c@entry=0x7f76b0000b10) at /build/ceph-13.0.2-1932-g458b4fb/src/osdc/Objecter.cc:1726
#11 0x00007f76d26e52e4 in Objecter::_scan_requests (this=this@entry=0x7f76c00aeb30, s=0x7f76c00af8a0, skipped_map=skipped_map@entry=false, cluster_full=cluster_full@entry=false, pool_full_map=0x7f76be7fb330, need_resend=..., need_resend_linger=..., need_resend_command=std::map with 0 elements, sul=...,
    gap_removed_snaps=0x7f76ac0016f8) at /build/ceph-13.0.2-1932-g458b4fb/src/osdc/Objecter.cc:1120
#12 0x00007f76d26eded5 in Objecter::handle_osd_map (this=this@entry=0x7f76c00aeb30, m=m@entry=0x7f76ac0014a0) at /build/ceph-13.0.2-1932-g458b4fb/src/osdc/Objecter.cc:1228

led to recursive lock of the session mutex (locked in _scan_requests,
and again in _finish_command).

Fix by making the callers for _finish_command (and
_check_command_map_dne) take the session lock.

Fixes: http://tracker.ceph.com/issues/23940
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21708 from dalgaaf/wip-da-SCA-20180425

Various fixes for SCA issues

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21690 from xiexingguo/wip-pr-20304

mon, osd: add create-time for pool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21659 from yangDL/master

pybind/ceph_argparse.py:'timeout' must in kwargs when call run_in_thread

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/thrash-old-clients: ms_type=simple

hammer does not support async messenger, so set ms_type to "simple" for
hammer client.

Fixes: http://tracker.ceph.com/issues/23922
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix linter complaints

In addition to line ordering, there were a couple of bogus ones:
E: 30, 0: No name 'version' in module 'distutils' (no-name-in-module)
E: 30, 0: Unable to import 'distutils.version' (import-error)
E: 36, 8: No name 'wsgiserver' in module 'cherrypy' (no-name-in-module)
E: 36, 8: Unable to import 'cherrypy.wsgiserver.wsgiserver2' (import-error)

I don't know why pylint can't see these modules, but they're definitely
there, so I've added them to the ignored list in .pylintrc

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #21733 from trociny/wip-23938

qa/workunits/rbd: potential race in mirror disconnect test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21671 from jan--f/mgr-module-config-doc

doc/mgr/plugins: add note about distinction between config and kv store

Reviewed-by: John Spray <john.spray@redhat.com>

qa/workunits/rbd: potential race in mirror disconnect test

(due to a typo in get_image_id command arg)

Fixes: http://tracker.ceph.com/issues/23938
Signed-off-by: Mykola Golub <mgolub@suse.com>

client: drop function _get_inodeno

Drop _get_inodeno() as per the comment in https://github.com/ceph/ceph/pull/21554.

Signed-off-by: Jos Collin <jcollin@redhat.com>

client: use common interp of st_nlink for dirs

Apparently some applications use this (like mail servers) and since it's
trivial to support, let's do it. Idea is that st_nlinks for a directory is
either 0 (it is unlinked) or 2 + the number of sub-directories (which have ..
parent links).

Fixes: https://tracker.ceph.com/issues/23873
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd/PG: fix DeferRecovery vs AllReplicasRecovered race

- DeferRecovery event queued by AsyncReserver due to preemption
  event.  We are in Recovering state with RECOVERING bit set.
- We finish recovery, clear RECOVERING state bit, and queue
  AllReplicasRecovered from PrimaryLogPG::start_recovery_ops()
- DeferRecovery event arrives, moving us from Recovering -> NotRecovering
- AllReplciasRecovered event arrives, crashing us.

This is all hard to deal with because the events are queued and may
arrive later.  Solve the problem here by tolerating a delayed
DeferRecovery event: if the RECOVERING pg state bit isn't set, ignore
it (it's old).  The async reserver cancel events are unpredictable.

Fixes: http://tracker.ceph.com/issues/23860
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #21554 into master

* refs/pull/21554/head:
client: avoid second lock on client_lock

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #21592 into master

* refs/pull/21592/head:
mds: filter out blacklisted clients when importing caps
mds: don't add blacklisted clients to reconnect gather set
mds: combine MDCache::{cap_exports,cap_export_targets}

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21593 into master

* refs/pull/21593/head:
mds: properly check auth subtree count in MDCache::shutdown_pass()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21601 into master

* refs/pull/21601/head:
mds: don't discover inode/dirfrag when mds is in 'starting' state

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21610 into master

* refs/pull/21610/head:
cephfs-journal-tool: wait prezero ops before destroying journal

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21625 into master

* refs/pull/21625/head:
qa: remove racy/buggy test_purge_queue_op_rate

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21664 into master

* refs/pull/21664/head:
mds: properly trim log segments after scrub repairs something

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21728 from yuriw/wip-yuriw-crontab

qa/tests: fixed typo

qa/tests: fixed typo

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Revert "ceph.spec: work around build.opensuse.org"

This reverts commit 21d941e83b168fa297aed58e27b4c11193468293 which introduced a
"kludge" to make 32-bit x86 builds work in the openSUSE Build Service (OBS).

The OBS no longer uses i386 in RPM_OPT_FLAGS when the i586 build target is
specified. The current value of RPM_OPT_FLAGS for i586 is:

-fomit-frame-pointer -fmessage-length=0 -grecord-gcc-switches -O2 -Wall
-D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables
-fasynchronous-unwind-tables -fstack-clash-protection -g

(Side note: we are not currently building Ceph for any 32-bit architectures
in the OBS, and there are no plans to start doing so. That doesn't mean it
won't ever happen, but even if it does, this "kludge" will not be needed.)

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Conflicts:
ceph.spec.in

Merge pull request #21583 from yanghonggang/disable-cache

kv/RocksDBStore: fix rocksdb error when block cache is disabled

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #21670 from xiexingguo/wip-23878

osd/OSDMap: more pg upmap fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21709 from liewegas/wip-23917

ceph_test_rados_api_aio: fix race with full pool and osdmap

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21641 from zy751713126/total_size_format

tools: correct total size formatting

Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind/ceph_argparse.py: argument 'target' change to func

'run_in_thread' first argument named 'target'.
It is confusing with cluster.mon_command last argument 'target=None'.
So change 'run_in_thread' and RadosThread 'target' to 'func'

Signed-off-by: yangdeliu <yangdeliu@szsandstone.com>

Merge pull request #21710 from neha-ojha/wip-cosbench-workloads

qa: add cosbench workloads and override teuthology default settings

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mon, osd: add create-time for pool

We want to know how old the pools currently are, on which mgr/balancer
can make some time-related smart decisions based.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/suites/perf-basic: add cosbench write workload

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/perf-basic: override default teuthology settings

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/rados/perf: add cosbench workloads

Signed-off-by: Neha Ojha <nojha@redhat.com>

ceph_test_rados_api_aio: fix race with full pool and osdmap

We send ops until we get a EDQUOT, and then assert our next op to
a different object also gets EDQUOT.  However, if the second osd
doesn't have as new a map it may succeed.  Make sure the client has
the latest (mon) map, and thus the one marking the pool full, before
we send the second op.  That ensures the second OSD also has that
newer map and also returns EDQUOT.

Fixes: http://tracker.ceph.com/issues/23917
Signed-off-by: Sage Weil <sage@redhat.com>

test/perf_local.cc: reduce scope of some variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

client/Client.cc: reduce scope of variable 'r'

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #21704 from liewegas/wip-rados-old-clients

qa/suites/rados/thrash-old-clients: add rbd tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21339 from guzhongyan/tp-cleanup

osd: drop unused osd_disk_tp related options

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21630 from neha-ojha/wip-rgw-pools

qa: modify rgw default pool names

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

common/TrackedOp: fix 'warned' handling

Fix for this cppcheck issue:

[src/common/TrackedOp.cc:385]: (style) The scope of the variable
 'warned' can be reduced.

Actually it seems that warned was not set correctly anymore
after rewrite of the code. Set 'warned' in with_slow_ops_in_flight()
as already done with number of slow iops.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

qa/suites/rados/thrash-old-clients: add cls_rbd, rbd api tests

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/perf: override default teuthology settings

Signed-off-by: Neha Ojha <nojha@redhat.com>

kv/RocksDBStore: fix rocksdb error when block cache is disabled

Fixes: http://tracker.ceph.com/issues/23816
Signed-off-by: Yang Honggang <yanghonggang@umcloud.com>

Merge pull request #21703 from liewegas/wip-rados-old-clients

qa/suites/rados/thrash-old-clients: add hammer clients

qa/suites/rados/thrash-old-clients: add hammer

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21679 from liewegas/wip-rados-old-clients

qa/suites/rados/thrash-old-clients: do some thrashing with jewel and luminous clients

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21605 from liewegas/wip-osdmap-encode

mon: cache reencoded osdmaps

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Xiaoxi CHEN <xiaoxchen@ebay.com>

messages/MOSDMap: fix compat_version for MOSDMap

Jewel clients see
 will not decode message of type 41 version 4 because compat_version 4 > supported version 3

The problem is when compat_version == 0 and head_version != 0, we
interpret compat_version == head_version.  From Message::encode,

    // if the encoder didn't specify past compatibility, we assume it
    // is incompatible.
    if (header.compat_version == 0)
      header.compat_version = header.version;

Broken by 49833c3bb264949b8126796997a95a95b50af411 for the
head_version=4 case (compat_version needs to be 3).

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: clean up member init

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDMap: significant feature bits.

1. MOSDMap go with features indicating the features
this map encoded for.

2. Only reencode if significant bits mismatch between
target features and my features.

also update mon/OSDMonitor and osd/OSDService to adopt this change.

Signed-off-by: Xiaoxi CHEN <xiaoxchen@ebay.com>

mon/OSDMonitor: add feature into osdmap cache key.

Change the cache from
SimpleLRU<version_t, bufferlist> inc_osd_cache;
to
SimpleLRU<pair<version_t, uint64_t>, bufferlist> inc_osd_cache;

By doing this , we can cache several encoded osdmap for different
features, not necessary to re-encoding for each client.

Signed-off-by: Xiaoxi CHEN <xiaoxchen@ebay.com>

osdc/Objector: use std::shared_mutex instead of boost::shared_mutex

Since boost::threads support interruption by default locking over
boost::shared_mutexes may throw if the thread is interrupted. Replacing this by
std:: variant

Fixes: http://tracker.ceph.com/issues/23910
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/dashboard: Be more selective when catching exceptions.

- Do some minor improvements in the RGW client
- Do not catch ALL exceptions to do not block the current exception handler that prints the stack trace in the log file
- Throw RequestException istead of Exception when user does not have the required keys

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Address review comments

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Add RGW management features

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Add Copy2ClipboardButtonDirective

Signed-off-by: Volker Theile <vtheile@suse.com>

qa: update dashboard tests for https://

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: enable SSL in standby mode

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: implement get_store in StandbyPyModules

Signed-off-by: John Spray <john.spray@redhat.com>