qa/workunits/rados/test_crash.sh: suppress core files

The cores will make teuthology fail the job--and we don't want them for
this test, where we are deliberately causing crashes.

Fixes: https://tracker.ceph.com/issues/43653
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32713 into master

* refs/pull/32713/head:
qa/tasks/cephadm: start watching initial daemons before bootstrap
qa/tasks/cephadm: create /etc/ceph if it doesn't exist
qa/tasks/cephadm: fix log whitelist when there is no whitelist

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32701 into master

* refs/pull/32701/head:
qa/tasks/ceph_manager: fix ceph-objectstore-tool calls
qa/tasks/ceph_manager: fix admin_socket remote when using cephadm
qa/tasks/ceph_manager: --no-mon-config to ceph-objectstore-tool
qa/tasks/ceph_manager: fix filestore split command
qa/tasks/ceph_manager: fix import line

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/ceph_manager: fix ceph-objectstore-tool calls

Pass the correct paths based on whether this is the importing or exporting
OSD.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: fix admin_socket remote when using cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: --no-mon-config to ceph-objectstore-tool

The config is currently fetched at osd.admin, so the keyring is not
found.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: fix filestore split command

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: fix import line

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: start watching initial daemons before bootstrap

This lets us see output while bootstrap is happening.

(Depends on the teuthology change to use journalctl, see
teuthology commit 4fa83040b05b604280789459f095d6f2ad1b0d01.)

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: create /etc/ceph if it doesn't exist

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: fix log whitelist when there is no whitelist

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32698 into master

* refs/pull/32698/head:
common: drop the unused mutable_item_history container.
common: drop the unused prune() of {safe,mutable}_item_history.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32663 from dzafman/wip-41016

osd: Diagnostic logging for upmap cleaning

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32298 from dang/wip-dang-zipper-4

Zipper - RGWUser

common: drop the unused mutable_item_history container.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: drop the unused prune() of {safe,mutable}_item_history.

The presence of these never-used methods obscure bug investigation.
For instance, `safe_item_history` is used to record all messenger's
addresses. As we never lock reading from it in `_conn_prefix()`, we
expect they live as long their corresponding messenger instances.
When such an item becomes corrupted, presence of `prune()` imposes
verification the method is never called – this effort could have
been avoided if we hadn't have the dangerous method.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Zipper - RGWUser

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #32624 from alimaredia/wip-s3-tests-python-3

qa/suites: use s3-tests with python3 support

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32650 from votdev/fix_i18n

mgr/dashboard: Fix missing i18n.

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #32367 from capri1989/doc-gc-update

doc/radosgw/config-ref: paragraph to explain the gc settings

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #32049 from mgfritch/wip-cd-timeout

cephadm: add ability to specify a timeout

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32691 from rosinL/wip-txpkt

msg/async/dpdk: Fix infinite loop when sending packets

Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/async/dpdk: Fix infinite loop when sending packets

tx_pkt_burst loops until the next pointer is null, set next pointer of
the last segment nullptr to fix this issue.

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Signed-off-by: luo rixin <luorixin@huawei.com>

Merge PR #32114 into master

* refs/pull/32114/head:
doc: split index.rst of dev guide
doc: tweak few section titles in dev guide
doc: don't display contents twice in dev guide
doc: add guide for running tests with teuthology

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32513 from IlsooByun/loop_rgw_none

rgw: move forward marker even in case of many rgw.none indexes

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #31878 from yuvalif/wip-yuval-tag-filter

rgw/pubsub: notification filtering by object tags

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #32596 from linuxbox2/wip-rgwfile-pfront

rgw_file: avoid string::front() on empty path

Merge pull request #32453 from majianpeng/kv-rocksdbstore-fix-option-work

kv/RocksDBStore: make option: compaction_threads/disableWAL/flusher_t…

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #32443 from tchaikov/wip-btree

include/cpp-btree: cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #32670 into master

* refs/pull/32670/head:
doc: remove prod cluster examples from hardware recs

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31923 from tspmelo/wip-vertical-navigation

mgr/dashboard: Add vertical menu

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #32654 from liewegas/fix-43273

mgr/rook: provide full path for devices names in inventory

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32372 from sebastian-philipp/cephadm-progress

mgr/cephadm: Add progress to update_mgr()

Reviewed-by: Joshua Schmid <jschmid@suse.de>

doc: remove prod cluster examples from hardware recs

This PR removes the now-very-old (2012) production cluster
examples from the hardware recommendations page.

Sage requested this change in IRC on 14 Jan 2020. There is no issue link.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #32497 from zhengchengyao/snap_exists

rbd: add snap_exists method API

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: Diagnostic logging for upmap cleaning

Fixes: https://tracker.ceph.com/issues/41016
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #32541 into master

* refs/pull/32541/head:
mds: add comment in feature_bitset_t::empty

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32619 into master

* refs/pull/32619/head:
kv/RocksDBStore: break out of compaction thread early on shutdown
kv/RocksDBStore: debug async compaction

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32569 from votdev/feature_toggle_output

mgr/dashboard: Cleanup feature toggle status output

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #32635 from Devp00l/wip-43594

mgr/dashbaord: Fix E2E pools page failure

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/rook: provide full path for devices names in inventory

This matches the cephadm behavior.

Fixes: https://tracker.ceph.com/issues/43273
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32411 into master

* refs/pull/32411/head:
qa/tasks/ceph_manager: enable ceph-objectstore-tool via cephadm
qa/tasks/cephadm: support crush_tunables config option

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/ceph_manager: enable ceph-objectstore-tool via cephadm

- drop support for keyvaluestore
- leave a few paths non-cephadm specific (filestore, upgrade workaround)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32643 into master

* refs/pull/32643/head:
doc/bootstrap: fixed default --keyring target

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32567 from trociny/wip-mirror-snap-promote

librbd: fix some edge cases for snapshot mirror mode promote

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Fix missing i18n.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashbaord: Fix E2E pools page failure

The problem was that pg autoscaling is set to "on" by default. Therefore
the pg_num field is hidden, which caused the error. Now the test sets
the pg autoscaling to "off" revealing the pg_num field to make the tests
pass.

Fixes: https://tracker.ceph.com/issues/43594
Fixes: https://tracker.ceph.com/issues/43102
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: PageHelper.selectOption

This helper method selects an option inside a select element.
It will also expect that the option was set.

Fixes: https://tracker.ceph.com/issues/43594
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #32584 from aclamk/fix-bluefs-vselector

bluestore/bluefs: make accounting resiliant to unlock()

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #32625 from mgfritch/cephadm-workunit-test-cleanup

qa/workunits/cephadm/test_cephadm: workunit test cleanup

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31763 into master

* refs/pull/31763/head:
test: auto-upgrade subvolume test
mgr/volumes: tie everything together to implement versioned subvolumes
mgr/volumes: provide subvolume create/remove/open APIs
mgr/volumes: implement subvolume based on subvolume template
mgr/volumes: implement subvolume group based on group template
mgr/volumes: implement trash as a subvolume group
mgr/volumes: snapshot util module
mgr/volumes: template for implementing groups and subvolumes
mgr/volumes: implement filesystem volume module
mgr/volumes: lock module to serialize volume operations
mgr/volumes: introduce volume specification module
mgr/volumes: add fs_util helper module

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

bluestore/bluefs: make accounting resiliant to unlock()

Moved vselector.sub_usage, so sub_usage and add_usage will not cross
_flush_bdev_safely(), which contains unlock().

Fixes: https://tracker.ceph.com/issues/43538
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #32615 from liewegas/fix-43580

osd/PG: clean up fastinfo key when last_update does not increase

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30985 from chenerqi/master

mgr: ceph fs status support json format

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

include/cpp-btree: use "constexpr inline" for constants

in C++17, it's a more general way to define constants. and these
constants are correctly typed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/common/test_interval_set: use ASSERT_EQ()

ASSERT_EQ() is better than ASSERT_TRUE() when checking for equality, as
it is able to print the unmatched variables.

also, in ASSERT_EQ(), the first parameter is the expected value. so
transpose the parameters for better error messages.

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/interval_set: use prefix operator for non-trivial iterator

for better performance. no need to create a temporary instance of
iterator and throw it away

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/cpp-btree: adapt from abseil::btree

adapt based on btree implementation from Abseil Common Libraries.

with following changes:

* remove abseil dependencies. as abseil libraries do support C++11,
  they are written in C++11 compatible way. and use its homebrew
  alternatives for the facilities offered by C++14 and C++17. but
  Ceph is in C++17, so replace those facilities using the ones
  from the C++ standard library
* change `key_compare_to_adapter` to adapt to arithmetic types, and
  those who have `compare()` defined.
* move SlotPolicy into *_params types, simpler this way.
* continue using *_fields types. the layout facilities is nice. but
  we don't need this flexibility at this moment.
* continue using `empty_base_handle` for EBCO, CompressedTuple is
  nice, but it involves more dependencies which we don't want at
  this moment.
* node handle support is dropped. it's part of C++17 standard for
  node-based containers. we don't use it at this moment. but it's
  relatively simple if we want to add it back.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32630 from tchaikov/wip-43306

common/util: use ifstream to read from /proc files

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

doc/bootstrap: fixed default --keyring target

Changed target from 'ceph.keyring' to 'ceph.client.admin.keyring'.

Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>

cpp-btree: import abseil-cpp btree implementation

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/osd_types: add pg_t::compare()

it's equivalent to the `<=>` operator in C++20. it helps to simplify the
definition of other equality operators.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #32315 into master

* refs/pull/32315/head:
client: disallow changing fuse_default_permissions option at runtime

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #32322 into master

* refs/pull/32322/head:
mds: Reorganize class members in SimpleLock header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32348 into master

* refs/pull/32348/head:
mds: Move data members of struct SnapRealm to private.
mds: Reorganize structure members in SnapRealm header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32416 into master

* refs/pull/32416/head:
mds: Reorganize structure members in flock header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32431 into master

* refs/pull/32431/head:
mds: Reorganize structure members in inode_backtrace header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: add snap_exists method API

Signed-off-by: Zheng Yin <zhengyin@cmss.chinamobile.com>

Merge PR #32585 into master

* refs/pull/32585/head:
MDSMonitor: handle standby already without fscid

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32505 into master

* refs/pull/32505/head:
mds: note client features when rejecting client
mds: add hex indicator to printed feature flags

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #32095 into master

* refs/pull/32095/head:
ceph-volume: raw: activate: drop --all, --osd-id and --osd-fsid args
raw: fix activate args
raw: --osd-fsid throughout
ceph-volume: add raw mode
ceph-volume: show devices with bluestore labels and unavailable

Reviewed-by: Sébastien Han <seb@redhat.com>

Merge PR #32582 into master

* refs/pull/32582/head:
qa/cephfs: don't run kclient tests on latest RHEL
qa/cephfs: don't test kclient on RHEL 7

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32620 from cbodley/wip-qa-rgw-hadoop-versions

qa/rgw: s3a-hadoop task defaults to maven-version 3.6.3

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #32492 from yaarith/wip-fix-device-anon

mgr/telemetry: fix device serial number anonymization

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32608 from tchaikov/wip-rpm-byte-compile-with-python3

ceph.spec.in: use python3 to bytecompile .py files

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32605 from tchaikov/wip-silence-clang-warnings

crimson: cleanups for clang build

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

kv/RocksDBStore: break out of compaction thread early on shutdown

Fixes: https://tracker.ceph.com/issues/43587
Signed-off-by: Sage Weil <sage@redhat.com>

kv/RocksDBStore: debug async compaction

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32622 into master

* refs/pull/32622/head:
qa/workunits/cephadm/test_cephadm: drop stray 'exit 0'

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/rgw: remove test against hadoop v2.8.5

the hadoop branch rel/release-2.8.5 fails to build with:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:37 min
[INFO] Finished at: 2020-01-14T13:09:02Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (create-parallel-tests-dirs) on project hadoop-aws: An Ant BuildException has occured: Unable to create javax script engine for javascript

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: remove test against hadoop trunk

hadoop's trunk fails to build with many 'package does not exist' errors:

[INFO] 100 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  48.792 s
[INFO] Finished at: 2020-01-14T13:07:36Z
[INFO] ------------------------------------------------------------------------

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/cephadm: add progress to update_mgr()

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32032 from votdev/issue_43088_passwd_async_validator

mgr/dashboard: Process password complexity checks immediately

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #32574 from sebastian-philipp/orchestrator-exception-add-type

mgr/orchestrator: raise_if_exception: Add exception type to message

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #32491 from pcuzner/cephadm-deploy-grafana

cephadm: Add ability to deploy grafana container

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

doc: split index.rst of dev guide

Move every section to a different file. Also, fix minor issues like
line-wrapping along.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

doc: tweak few section titles in dev guide

Signed-off-by: Rishabh Dave <ridave@redhat.com>

doc: don't display contents twice in dev guide

Content of this page is being displayed twice, once on the left side of
the page and once within the page. Display it only once at the left
side of the page.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

doc: add guide for running tests with teuthology

The guide shows how to run integration tests using teuthology framework.

Fixes: https://tracker.ceph.com/issues/42749
Signed-off-by: Rishabh Dave <ridave@redhat.com>

common/util: use ifstream to read from /proc files

refactor `collect_sys_info()` to use ifstream, getline and boost string
helpers to read and parse the /proc/meminfo and /proc/cpuinfo. for
better readability, and it's less error-prone.

Related-To: https://tracker.ceph.com/issues/43306
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29669 from cqzcqq/fix-40117

osd/PeeringState.h: Fix pg stuck in WaitActingChange

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PG: clean up fastinfo key when last_update does not increase

If the last_update field goes backwards, we write a full info instead of
the fastinfo key. However, a stale fastinfo key may still be present, and
it may have a last_update that looks valid to a restarting OSD.

Fixes: https://tracker.ceph.com/issues/43580
Reported-by: song <jinglesong0814@163.com>
Signed-off-by: Sage Weil <sage@redhat.com>

crimson: move PG::prepare_write() to .cc file

less cluttered this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

use omap_rmkey() where appropriate

Signed-off-by: Sage Weil <sage@redhat.com>

os/Transaction: add omap_rmkey

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32595 from rs-fabrica/doc_install_upgrading-ceph_systemctl_use

doc/install/upgrading-ceph: systemctl in Ubuntu instructions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32607 from tchaikov/wip-odr-global-context

cmake: do not include global_context.cc multiple times

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/tasks/cephadm: support crush_tunables config option

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31917 into master

* refs/pull/31917/head:
vstart: Remove duplicate option -N

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume: raw: activate: drop --all, --osd-id and --osd-fsid args

We don't need these currently.

Signed-off-by: Sage Weil <sage@redhat.com>