Merge pull request #22867 from smithfarm/wip-ccache-tweak

run-make-check.sh: ccache goodness for everyone

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge remote-tracking branch 'gh/master'

Merge remote-tracking branch 'private/wip-cx'

 Conflicts:
src/include/ceph_features.h
src/msg/async/AsyncConnection.cc
src/msg/async/AsyncConnection.h

- SERVER_NAUTLIUS feature recently added to default
- msgr2 bool
- ceph::shared_ptr -> std::shared_ptr

Merge pull request #22778 from trociny/wip-mock-warnings

test/librbd: fix gmock warning in TestMockIoImageRequestWQ.AcquireLockError

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22903 from tspmelo/fix-e2e-script

mgr/dashboard: Fix e2e script

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

run-make-check.sh: fill in missing DRY_RUN and add doc hint

Signed-off-by: Nathan Cutler <ncutler@suse.com>

run-make-check.sh: reset trap after sourcing install-deps.sh

Since we source install-deps.sh, it replaces our trap with its own,
so we have to reinstate our trap after it finishes.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

cleanup: run-make-check.sh: no space between -D and name of cmake option

Signed-off-by: Nathan Cutler <ncutler@suse.com>

run-make-check.sh: gtest-parallel only with Python 2

Fixes: http://tracker.ceph.com/issues/24817
Signed-off-by: Nathan Cutler <ncutler@suse.com>

run-make-check.sh: inform user of make -j setting

This can help the user figure out why the build is so slow.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: cleanup: wordsmith a comment in run-make-check.sh

Signed-off-by: Nathan Cutler <ncutler@suse.com>

run-make-check.sh: ccache goodness for everyone

Since run-make-check.sh already ensures that ccache is installed,
it makes sense to let everyone benefit from the ccache
tweaks introduced by 4cb5a590537a9caaf61db42ce8ea123d2ab961f3

Note 1: The previous solution using "date" would cause build tools to reset
their timestamps after 24 hours, on subsequent runs of run-make-check.sh.
In order to maximize ccache effectiveness, this commit sets SOURCE_DATE_EPOCH
to a fixed value: the number of seconds elapsed since the Unix epoch as at
January 1, 2000 (chosen to commemorate Y2K armageddon).

Note 2: this commit introduces "set -e". This was actually in effect
before, via "source install-deps.sh". Better to make it explicit.

Fixes: http://tracker.ceph.com/issues/24777
Signed-off-by: Nathan Cutler <ncutler@suse.com>

run-make-check.sh: drop workaround for undeclared jq build dependency

jq build dependency is properly declared now.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #22898 from majianpeng/bluestore-remove-redundant-fault-range

os/bluestore: remove redundant fault_range.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22893 from jcsp/wip-misc

mgr: Miscellaneous small mgr fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21962 from socketpair/perf

osd: Additional fields for osd "bench" command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #22836 into master

* refs/pull/22836/head:
qa: remove unused distro dir

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge PR #22741 into master

* refs/pull/22741/head:
qa: fix ffsb cd argument
qa: fix cd argument during iogen unpack

Reviewed-by: Neha Ojha <nojha@redhat.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22766 into master

* refs/pull/22766/head:
mds: refactor const ref return to avoid mem alloc

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22740 into master

* refs/pull/22740/head:
qa: create common conf for all cephfs suites
qa: remove wrongly created random distro conf

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #22922 from smithfarm/wip-argparse-py3

tests: make test_ceph_argparse.py pass on py3-only systems

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22912 from tchaikov/wip-cmake-cleanup

cmake: remove workarounds for supporting cmake 2.x

Reviewed-by: Casey Bodley <cbodley@redhat.com>

tests: make test_ceph_argparse.py pass on py3-only systems

On systems that do not have Python 2 installed, this test fails to
import StringIO.

Fixes: http://tracker.ceph.com/issues/24816
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #22863 into master

* refs/pull/22863/head:
rgw: fix gc may cause a large number of read traffic

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20779 from oritwas/wip-rgw-23257

rgw: radogw-admin reshard status command should print text for reshard status

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #22721 from linuxbox2/wip-rgw-23099

rgw: escape markers in RGWOp_Metadata_List::execute

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #22745 into master

* refs/pull/22745/head:
debian/control: require sudo
ceph.spec: require sudo
ceph.spec: install etc/sudoers.d/ceph-osd-smartctl
debian: install etc/sudoers.d/ceph-osd-smartctl
sudoers.d/ceph-osd-smartctl: allow user ceph to run smartctl as root

Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge PR #22858 into master

* refs/pull/22858/head:
osd/PrimaryLogPG: rebuild attrs from clients

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #22870 into master

* refs/pull/22870/head:
mgr: replace iteritems with six.iteritems (mgr_module.py)
mgr: replace iteritems with six.iteritems (crash)
mgr: replace iteritems with six.iteritems (telegraf)
mgr: replace iteritems with six.iteritems (status)
mgr: replace iteritems with six.iteritems (restful)
mgr: replace iteritems with six.iteritems (influx)
mgr: replace iteritems with six.iteritems (balancer)

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #22871 into master

* refs/pull/22871/head:
deb/rpm: add python-six as build-time and run-time dependency

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #22873 into master

* refs/pull/22873/head:
os/bluestore: fix incomplete faulty range marking when doing compression

Reviewed-by: Sage Weil <sage@redhat.com>

cmake: remove workarounds for supporting cmake 2.x

and bump up the required cmake version to 3.5.1

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22896 from tchaikov/wip-cmake3

deb,rpm,do_cmake: switch to cmake3

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

os/bluestore: remove redundant fault_range.

In fact, _do_read already did fault_range for the same offset and
length.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #22619 from zhangsw/fix-rgw-deletemarker-bug

rgw: The delete markers generated by object expiration should have owner attribute.

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #22145 from Jeegn-Chen/wip-issue-24223

rgw: reject invalid methods in validate_cors_rule_method

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22452 from theanalyst/wip-placement-target-warn

rgw: warn if zone doesn't contain all zg's placement targets

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #22614 from tianshan/fix_24563

rgw: fix copy response header etag format not correct

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #22623 from zhangsw/fix-rgw-lifecycle-rule-num

rgw: Limit the number of lifecycle rules on one bucket.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #22635 from zhangsw/fix-rgw-admin-objectexpire-bug

rgw: "radosgw-admin objects expire" always returns ok even if the process fails.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

mgr/dashboard: Fix e2e script

Angular 6 changed the cli and how you can configure the e2e cmd.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr: reduce log spam

Several things at level 1 that shouldn't have been, especially
the big dump of all possible commands every time we handle one.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: handle long-named mgr modules

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: improve logging on handle_command path

Signed-off-by: John Spray <john.spray@redhat.com>

deb,rpm,do_cmake: switch to cmake3

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian/control: require sudo

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: require sudo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22878 from falcon78921/wip-falcon78921-docs-install

doc: fixed minor typo in Debian packages section

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: fixed minor typo in Debian packages section

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

mgr: replace iteritems with six.iteritems (mgr_module.py)

Fixes: http://tracker.ceph.com/issues/24779
Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr: replace iteritems with six.iteritems (crash)

Fixes: http://tracker.ceph.com/issues/24779
Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr: replace iteritems with six.iteritems (telegraf)

Fixes: http://tracker.ceph.com/issues/24779
Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr: replace iteritems with six.iteritems (status)

Fixes: http://tracker.ceph.com/issues/24779
Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr: replace iteritems with six.iteritems (restful)

Fixes: http://tracker.ceph.com/issues/24779
Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr: replace iteritems with six.iteritems (influx)

Fixes: http://tracker.ceph.com/issues/24779
Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr: replace iteritems with six.iteritems (balancer)

Fixes: http://tracker.ceph.com/issues/24779
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #22874 from ddiss/bluestore_docs

doc/bluestore: fix minor typos in compression section

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: fix incomplete faulty range marking when doing compression
GC.

Under some scenarios GC might process an extent range where some inner extents are left untouched by GC (as there is no need for that). Hence GC doesn't invaliate these inner extents with fault_range call. If untouched extents are mapped to unloaded shards it results in subsequent assertion on o->extent_map.dirty_range() call.
The solution is to invalidate the whole extent range when doing GC.

Fixes: https://tracker.ceph.com/issues/23540
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #22617 from tianshan/fix_http_err_return

rgw: set cr state if aio_read err return in RGWCloneMetaLogCoroutine

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22793 from kshtsk/wip-add-opensuse-leap

Add new OpenSUSE Leap id for install-deps.sh

Reviewed-by: Nathan Cutler <ncutler@suse.com>

rgw: radogw-admin reshard status command should print text for reshard status

Fixes: http://tracker.ceph.com/issues/23257
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

doc/bluestore: fix minor typos in compression section

Signed-off-by: David Disseldorp <ddiss@suse.de>

Merge pull request #22829 from tspmelo/fix-24757

mgr/dashboard: Fix regression on rbd form component

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #22762 from ricardoasmarques/wip-toolbar-text

mgr/dashboard: Remove top-right actions text and add "About" page

Reviewed-by: Kanika Murarka <murarkakanika@gmail.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

deb/rpm: add python-six as build-time and run-time dependency

python-six is used by dashboard's controllers. please note,
${python:Depends} is able to fill the installation dependencies for
debian package. so python-six is not added to Requires of ceph-mgr.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20133 from smithfarm/wip-redundant-recommends

build/ops: deb: drop redundant ceph-common recommends

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: fix gc may cause a large number of read traffic

fix http://tracker.ceph.com/issues/24767

Signed-off-by: Xin Liao <liaoxin01@baidu.com>

Merge pull request #22847 from ErwanAliasr1/evelu-ccache2

run-make-check.sh: Adding ccache tuning for the CI

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: create common conf for all cephfs suites

This will be followed by removing common CephFS configurations in the
ceph.conf.template in teuthology.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22703 into master

* refs/pull/22703/head:
doc/mgr: add doc for crash mgr module
qa/suites/rados/mgr: Add test_crash
qa/tasks/mgr: add test_crash, call from test_module_selftest
qa/tasks/{ceph_manager.py,vstart_runner.py}: allow kwargs in raw_*
mgr/pybind/crash: handle crashdumps

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #22704 into master

* refs/pull/22704/head:
osd/PG: do not send notify to empty peer
osd/PG: do not assume delete event means found_missing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #22733 into master

* refs/pull/22733/head:
os/bluestore: set correctly shard for existed Collection.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #22738 into master

* refs/pull/22738/head:
mgr/status: add exabytes unit

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #22534 into master

* refs/pull/22534/head:
msg/Messenger: use safe_item_history<> for my_addrs
msg/simple: fix set_myaddrs()
msg/Messenger: use mutable_item_history<> for my_addrs
common/item_history: container to support lockless reads, infrequent updates
msg/simple/Accepter: fix my_addr == my_addrs.v[0] constraint
msg/async: track target_addr for each connection
msg/async: mark accepted connections with addr type (legacy or msgr2)
msg/async: mark AsyncConnection with msgr2 flag
msg/async: track connections by addrvec
mon/Session: inst -> name and addrs
osd/OSDMap: don't print hb addrs
msg/DispatchQueue: myaddr -> myaddrs
mgr: myaddr -> myaddrs
msg: make set_addr_unknowns take an addrvec
mon/LogMonitor: myaddr -> myaddrs
librados: myaddr -> myaddrs
common/LogClient: myaddr -> myaddrs
client: myaddr -> myaddrs
osd/OSDMap: is_blacklisted() for addrvecs
osd: populate metadata with all addrs
mds: addr -> addrvec
mon/MonClient: get_myaddrs()
mon/OSDMonitor: addrvec blacklist helper
mds: use new pick_addresses
ceph-osd: use new bindv()
msg/async: bind to multiple addresses
msg/async: (legacy) handshake using legacy addr
msg/async: fix some debug prefixes
msg/async: multiple listening ServerSockets
osd/OSDMap: make cluster addrs addrvecs too
msg/Messenger: my_addr -> my_addrs
msg/Connection: peer_addr -> peer_addrs
msg/msg_types: hash<> for entity_addrvec_t
mgr/DaemonServer: use new bindv() and pick_addresses()
msg/Messenger: new bindv() that takes an addrvec
common/pick_address: fill in ipv4/6 and msgr1/msgr2 via config options
common/pick_addresses: new addrvec-based pick_addresses()
common/pick_addresses: add filtering by ipv4 and ipv6
CEPH_MON_PORT -> CEPH_MON_PORT_LEGACY; define CEPH_MON_PORT_IANA
osd: kill osd_heartbeat_addr option
common/options: add addrvec option type

Reviewed-by: Ricardo Dias <rdias@suse.com>

osd/PrimaryLogPG: rebuild attrs from clients

Ensure that buffers coming in via client ops are rebuilt before populating
the PGTransaction.  This ensures that we don't pin the raw buffers for
the entire incoming message in memory.

In the past we've addressed this issue at the ObjectStore layer, but we
did not consider the attr_cache in ObjectContext.  Rebuilding the buffers
at this point will sanitize any incoming attribute before it reaches
either PGBackend or ObjectContext object_cache or the ObjectStore
implementation.

Fixes: http://tracker.ceph.com/issues/24768
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Fix regression on rbd form component

'decodeURIComponent' method was decoding an undefined variable to the string
'undefined', which would later cause problems.

This regression was introduced in057d6025894800c1b4a422b081fe02ab82b044fe.

Fixes: http://tracker.ceph.com/issues/24757
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22852 from ceph/jcsp-cephfs-exp-doc

doc/cephfs: remove lingering "experimental" note about multimds

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc/cephfs: remove lingering "experimental" note about multimds

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #22830 from ricardoasmarques/fix-24756

mgr/dashboard: Fix RBD object size dropdown options

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

run-make-check.sh: Adding ccache tuning for the CI

When run-make-check is run by the CI, some tuning can be performed to
speedup the build.

This commit :
- Detect jenkins by searching JENKINS_HOME env variable
- Defines the SOURCE_DATE_EPOCH to enforce a stable date across builds
- Ask cmake not to use the git versioning which adds useless entropy for a temporary build (ENABLE_GIT_VERSION=OFF)
- Define the ccache slopiness to increase efficiency
- Increase the ccache size to save multiple builds to maximise cache hit between PRs
- Print ccache statistics to evaluate ccache efficiency

Signed-off-by: Erwan Velu <erwan@redhat.com>

Merge pull request #22644 from Devp00l/wip-forms-helper-service

 mgr/dashboard: CdFormGroup

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix RBD object size dropdown options

Fixes: https://tracker.ceph.com/issues/24756
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #22461 from tspmelo/wip-osd-flags-ui

mgr/dashboard: Add UI for Cluster-wide OSD Flags configuration

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Add "About" modal

Fixes: https://tracker.ceph.com/issues/24646
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Usage of 'BehaviorSubject' in summary

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Remove text from top-right actions

Fixes: https://tracker.ceph.com/issues/24624
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #22757 from votdev/improve_str_to_bool

mgr/dashboard: Improve str_to_bool

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

os/bluestore: set correctly shard for existed Collection.

For existed Collection, the constructor of Collection will be called in _open_collections.
But m_finisher_num can't setup when enable bluestore_shard_finishers.

So move m_finisher_num setup before _open_collections && _kv_start.

Fixes: http://tracker.ceph.com/issues/24761
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

qa: fix ffsb cd argument

Avoid potential failures relating to HEAD^.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: fix cd argument during iogen unpack

Fixes: http://tracker.ceph.com/issues/24680
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22837 from dzafman/wip-fix-timeout

test: test_get_timeout_delays() fix

Reviewed-by: Erwan Velu <erwan@redhat.com>

test: test_get_timeout_delays() fix

Caused by: 7b0d1c8b8acff2a7010bfb0400df09786033ac63

Signed-off-by: David Zafman <dzafman@redhat.com>

msg/Messenger: use safe_item_history<> for my_addrs

Mutating the item in place is not safe.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/simple: fix set_myaddrs()

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Messenger: use mutable_item_history<> for my_addrs

We read this without a lock.

Signed-off-by: Sage Weil <sage@redhat.com>

common/item_history: container to support lockless reads, infrequent updates

Signed-off-by: Sage Weil <sage@redhat.com>

msg/simple/Accepter: fix my_addr == my_addrs.v[0] constraint

Signed-off-by: Sage Weil <sage@redhat.com>