doc/releases/nautilus: fix release notes (crash->device)

Fix pending note for octopus too

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31827 into master

* refs/pull/31827/head:
mgr/ssh: add 'upgrade check' command
ceph-daemon: add 'pull' command
mgr/orchestrator_cli: document orchestrator option
mgr/ssh: move entity name munging into a helper

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31970 into master

* refs/pull/31970/head:
doc/releases: add Nautilus 14.2.5 to release timeline
doc/releases: add release notes for nautilus v14.2.5

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32141 from tchaikov/wip-doc-highlight-console

doc: use the console lexer for rendering command line sessions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31965 from rosinL/wip-osd-support-bond-network

osd/osd: Refactor get_iface_numa_node

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27392 from rouming/bluestore-iouring

os/bluestore: introduce new io_uring IO engine

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31979 from jan--f/c-v-patch-volumegroups

ceph-volume/test: patch VolumeGroups

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32122 from s0nea/wip-dashboard-fix-tests

qa/tasks/mgr/dashboard/test_health: update schema

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

os/bluestore: introduce new io_uring IO engine

This implements low-level IO engine, which utilizes brand-new
io_uring IO interface: https://lwn.net/Articles/776428/

By default libaio is used.  If bluestore_ioring=true is set but kernel
does not support io_uring or architecture is not x86-64, libaio will be
used instead.

In current patch liburing library is used in order not to open code
everything.

In order to compile with liburing WITH_LIBURING=ON should be specified.

Signed-off-by: Roman Penyaev <rpenyaev@suse.de>

os/bluestore: make abstract interface for any sorts of AIO queues

In the next patch new io_uring API will be used instead of libaio.
So this prepares the abstract interface.

Signed-off-by: Roman Penyaev <rpenyaev@suse.de>

Merge pull request #32007 from tchaikov/wip-osd-cleanup

osd: use unique_ptr for managing life cycles

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #31931 from majianpeng/bluestore-bugfix

os/bluestore: fix wakeup bug

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #32052 from mgfritch/wip-cd-standalone-tempfiles

ceph-daemon: clean-up tempfiles on EXIT

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #32100 from yehudasa/wip-43184

qa: radosgw-admin: remove dependency on bunch package

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31961 from joscollin/wip-B43038-subvolume-group-create-fail

mgr/volumes: check for string values in uid/gid

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: use console lexer for rendering command line sessions

by default the pygements uses python lexer for rendering literal blocks,
and `#` is considered as the marker for inline comments. in this change,
the "console" lexer is specified explicitly. see
https://pygments.org/docs/lexers/#lexers-for-various-shells for more details,
in which "BashSessionLexer" is for this very use case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #32124 into master

* refs/pull/32124/head:
doc/cephfs/disaster-recovery-experts: Add link for scrub and note for scrub_path

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27894 into master

* refs/pull/27894/head:
cephfs-shell: Add default list to df
cephfs-shell: Add tests for df command
cephfs-shell: Fix 'df' command errors

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32131 into master

* refs/pull/32131/head:
codeowners: Add ceph2.py to @ceph/orchestrators

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32038 into master

* refs/pull/32038/head:
doc: reorganize CephFS landing page and ToC

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #32098 into master

* refs/pull/32098/head:
ceph-daemon: py2: tolerate whitespace before config key name

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #32037 from smithfarm/wip-43152-workaround

build/ops: make-dist: default to no dashboard frontend build parallelism

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

doc/releases: add Nautilus 14.2.5 to release timeline

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/releases: add release notes for nautilus v14.2.5

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #32088 from idryomov/wip-krbd-exclusive-lock-fixups

qa: krbd_exclusive_option.sh: update for recent kernel changes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32046 into master

* refs/pull/32046/head:
mgr/DaemonServer: fix 'osd ok-to-stop' for EC pools

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31489 into master

* refs/pull/31489/head:
pybind/rados: add WriteOp::writesame() and test WriteOp::writesame()
pybind/rados: add Ioctx::aio_writesame() and test Ioctx::aio_writesame()
pybind/rados: add Ioctx::writesame() and test Ioctx::writesame()

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge PR #29214 into master

* refs/pull/29214/head:
mgr: templatize/generalize metrics collection interface

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32102 into master

* refs/pull/32102/head:
mgr/diskprediction_local: import scipy early to fix self-test deadlock

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31794 from dillaman/wip-42889

librbd: disable zero-copy writes by default

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #32105 from junxfl/test1

librbd: fix rbd_open_by_id, rbd_open_by_id_read_only

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32109 from yehudasa/wip-42868

rgw: list buckets: don't return buckets if limit=0

codeowners: Add ceph2.py to @ceph/orchestrators

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32053 from votdev/left_align_badges

mgr/dashboard: Left align badge datatable columns

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #31927 from bk201/wip-43072

mgr/dashboard: hide in-use devices when creating OSDs

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #32084 from votdev/auth_service_observable

mgr/dashboard: Use Observable in auth.service

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

mgr/dashboard: Improve badge column template. (#32026)

mgr/dashboard: Improve badge column template.

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge PR #31907 into master

* refs/pull/31907/head:
mgr/dashboard: properly handle a missing rbd-mirror service status
mgr: cull service daemons when the last instance has been removed
mgr: ensure new daemons are properly indexed by hostname

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

qa: kernel.sh: unlock before rolling back

"rbd snap rollback" expects an unlocked image, but we may get there
locked if object map is enabled (or if lock_on_read is specified in
rbd_default_map_options).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa: krbd_exclusive_option.sh: update for recent kernel changes

Since 5.3:
- a plain "rbd map" acquires the lock, so it's not different from
  "rbd map -o exclusive" in this regard
- if the lock is held by the exclusive peer, I/O is failed right away
  instead of blocking
- lock_timeout option is respected only by "rbd map" and not by I/O

Since 5.5:
- if the mapping is read-only, the lock isn't acquired

Added blacklisting test case, dropped lock_timeout test case.

Fixes: https://tracker.ceph.com/issues/43127
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/tasks/mgr/dashboard/test_health: update schema

Fixes: https://tracker.ceph.com/issues/43195
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge PR #32009 into master

* refs/pull/32009/head:
doc: removes kube-helm installation instructions

Reviewed-by: Sebastian Wagner <swagner@suse.com>

librbd: fix rbd_open_by_id, rbd_open_by_id_read_only

These methods incorrectly delete ImageCtx on error, resulting
in double-free heap corruption.

Fixes: https://tracker.ceph.com/issues/43178
Signed-off-by: yangjun <yangjun@cmss.chinamobile.com>

mgr/volumes: check for string values in uid/gid

chown allows strings as per bbbfb44453f204286a8ff312349d42afde5fced6,
which caused this error. Eventhough uid/gid are input as CephInt, the
qa tests can pass in only string values to _fs_cmd. So try converting
the incoming uid/gid to int in create_subvolume and create_group.
It might be a valid string.

Fixes: https://tracker.ceph.com/issues/43038
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge PR #31800 into master

* refs/pull/31800/head:
doc: Update commands in bootstrap.rst
Update doc/bootstrap.rst
Update doc/bootstrap.rst
Merge branch 'master' into wip-doc-prompts-update
doc: Update commands in bootstrap.rst

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

ceph-daemon: py2: tolerate whitespace before config key name

The py2 ConfigParser doesn't like whitespace before the config option
name.  (The py3 version doesn't care.)  Filter it out before parsing.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephfs/disaster-recovery-experts: Add link for scrub and note for scrub_path

Fixes: https://tracker.ceph.com/issues/22204
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #32106 from taodd/ceph-volume-fix-env

ceph-volume: fix the integer overflow

mgr/dashboard: add smartctl data as a details tab of a host (#30594)

mgr/dashboard: add smartctl data as a details tab of a host

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

qa: radosgw-admin: remove dependency on bunch package

Fixes: https://tracker.ceph.com/issues/43184
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #29687 into master

* refs/pull/29687/head:
test/store_test: test coverage for anti-spillover framework.
os/bluestore: introduce bluestore_volume_selection_reserved cfg option.
os/bluestore: beautify RocksDBBlueFSVolumeSelector::dump output
os/bluestore: introduce 'bluestore bluefs stats' admin socket command
os/bluestore: implement more intelligent DB volume space managment for BlueFS.
os/kv: add static method to parse RocksDB options
os/bluestore: introduce volume selector abstraction to BlueFS

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32108 into master

* refs/pull/32108/head:
ceph-daemon: remove prepare-host

Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph-daemon: remove prepare-host

I thought I took this out of the PR but somehow it got merged in... must
have repushed and old branch and not realized.  :/

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31959 into master

* refs/pull/31959/head:
os/bluestore/BlueStore.cc: set priorities for compression stats

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #31995 into master

* refs/pull/31995/head:
msg/async: No need lock for func _filter_addrs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32101 into master

* refs/pull/32101/head:
mgr/orchestrator_cli: make rgw rm take realm+zone
qa/tasks/mgr/test_orchestrator_cli: fix rgw add test

Reviewed-by: Sebastian Wagner <swagner@suse.com>

rgw: list buckets: don't return buckets if limit=0

Fixes: https://tracker.ceph.com/issues/42868
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/orchestrator_cli: make rgw rm take realm+zone

And adjust test accordingly

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: fix the type mismatch, covert the tries and interval to int from string.
Fixes: https://tracker.ceph.com/issues/43186
Signed-off-by: dongdong tao <dongdong.tao@canonical.com>

mgr/diskprediction_local: import scipy early to fix self-test deadlock

We are seeing a hang on centos7 (but not ubuntu 18.04) from

        from .predictor import get_diskfailurepredictor_path

in _predict_life_expentancy.

Doing that same predictor import at the top of the file doesn't help--it
hangs right when the diskpredictor_local module is loaded.  Commenting
out the import scipy in predictor.py avoids the hang.

I'm not sure why, but doing the full scipy import here appears to work
around the problem.

Fixes: https://tracker.ceph.com/issues/42764
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32043 into master

* refs/pull/32043/head:
os/bluestore: default bluestore_block_size 1T -> 100G

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #32044 into master

* refs/pull/32044/head:
mon/OSDMonitor: make memory autotune disable itself if no rocksdb

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31581 into master

* refs/pull/31581/head:
mon: fix mon_sync_max_payload_size type
mon: cap keys in mon_sync messages
mon/MonitorDBStore: improve get_chunk_tx limits
mon/MonitorDBStore: better size estimation for Transaction

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32048 into master

* refs/pull/32048/head:
mon/PGMap: use NONE for 'pg ls[-*]' output too

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32039 into master

* refs/pull/32039/head:
test: Improve races by using kill_daemons which waits for OSDs terminate
test: run-standalone.sh: Only run execs in the subdirectories of qa/standalone
test: Use activate_osd() when restarting OSDs
test: osd-scrub-snaps.sh: Fix race with osd restart and doing a scrub

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: fix rgw add test

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32010 into master

* refs/pull/32010/head:
mgr/ssh: Add module to static type checking from tox.
mgr/orchestrator: Add mypy static type checking

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32023 into master

* refs/pull/32023/head:
qa/tasks/mgr/test_orchestrator_cli: test with valid network or IP
qa/orchestrator: do not test "mon update 3 host1"

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>

test: Improve races by using kill_daemons which waits for OSDs terminate

osd-backfill-space.sh: More sleep time to make sure the backfill gets started

Signed-off-by: David Zafman <dzafman@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: test with valid network or IP

Passing 'network' will fail with

Error EINVAL: Failed to parse host list: '['host1:network', 'host2:network', 'host3:network']': u'network' does not appear to be an IPv4 or IPv6 address

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32096 into master

* refs/pull/32096/head:
ceph-daemon: fix firewalld error case

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #32097 into master

* refs/pull/32097/head:
ceph-daemon: fix bootstrap ownership of tmp monmap file

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #32051 into master

* refs/pull/32051/head:
mgr: simply exit on SIGINT or SIGTERM

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-daemon: fix bootstrap ownership of tmp monmap file

The file is created with mode 0644 by the ceph:ceph user, but root
cannot write to that.  Instead, create the tmp file owned by root, write
to it, then chown it to user ceph.

Fixes e92929d86dce2297d3e02ec88c6830224aefc061

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: fix firewalld error case

Signed-off-by: Sage Weil <sage@redhat.com>

doc: reorganize CephFS landing page and ToC

Fixes: https://tracker.ceph.com/issues/42016
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32021 from falcon78921/wip-doc-43130

doc: fixed broken reference link for Graphviz

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32017 from dzafman/wip-revert-verify-upmap

Revert "crush: remove invalid upmap items"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

test: run-standalone.sh: Only run execs in the subdirectories of qa/standalone

This will ignore scripts placed at the qa/standalone level, though
I'm not sure if we should be putting any tests there.  It does
allow support scripts present like ceph-helper.sh without modifying
run-standalone.sh to ignore it.

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #31589 from zhengchengyao/mv_snapshot

librbd:move all snapshot API functions in internal.cc over to api/Snapshot.cc

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31767 from pritha-srivastava/wip-rgw-mfa-set-versioning

rgw: adding mfa code validation when bucket versioning status is changed

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

mgr/dashboard: adapt to create_osds parameters change (#31928)

mgr/dashboard: adapt to create_osds parameters change

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard: add smart data to hosts page

Fixes: https://tracker.ceph.com/issues/42064
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/dashboard: Use Observable in auth.service

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #31062 from dillaman/wip-42355

mgr/dashboard: support RBD mirroring bootstrap create/import

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: Cleanup code (#32031)

mgr/dashboard: Cleanup code

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #31949 from dillaman/wip-42748

rbd: incorporate rbd-mirror daemon status in mirror pool status

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Left align badge datatable columns

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge PR #29964 into master

* refs/pull/29964/head:
qa/cephfs: test conf file reading for CephFS shell
qa/cephfs: allow passing conf to methods that runs shell command
qa/vstart_runner: add a method to create temporary file
test_cephfs_shell: update test since shell can now read ceph.conf
src/vstart.sh: add cephfs-shell section to ceph.conf
cephfs-shell: read shell variables from ceph.conf

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-daemon: remove guesswork to find script file

Allow passing CEPH_DAEMON via the environment or default to using the
script from the standard location.

Signed-off-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: trap on EXIT

tempfiles were not being removed after a standalone test failure

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr: templatize/generalize metrics collection interface

templatize metrics collection so as to reuse quering routines.
`MetricCollector` can be subclassed and along with implementing
` process_reports()` to process incoming metrics data.

also, generalize metrics data in `MMgrReport` and metric query
configuration in `MMgrConfigure`.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa/cephfs: test conf file reading for CephFS shell

Adds tests that checks -
* reading conf option
* reading conf option after setting it twice
* reading conf option after setting it after a reset

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: allow passing conf to methods that runs shell command

Add an argument to the methods that issues CephFS shell command to
accept path to the ceph.conf file.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/vstart_runner: add a method to create temporary file

The method's signature is kept identical to the method in teuthology.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: update test since shell can now read ceph.conf

In the test
test_cephfs_shell.TestMisc.test_issue_cephfs_shell_cmd_at_invocation,
don't set the shell option with in the command to be tested since
CephFS shell can now read it's value from ceph.conf.

Also, add methods to set config options for vstart as well teuthology
testing.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

src/vstart.sh: add cephfs-shell section to ceph.conf

And set "debug" to true, so that developers can see tracebacks in case
cephfs-shell crashes.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: read shell variables from ceph.conf

Read ceph.conf and initialize shell variables by reading options set
under section cephfs-shell. If there are sections/options that are
duplicated in ceph.conf, CephFS shell overrides the options with the
values last.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge PR #31897 into master

* refs/pull/31897/head:
doc: obsolete entries for allow_standby_replay
wip-doc: entries obsoleted by allow_standby_replay

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31403 into master

* refs/pull/31403/head:
vstart.sh: highlight presence of stray conf

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>