test/pybind/rbd.pyx: add test_remove_snap_by_id case in test_rbd.py

Signed-off-by: Zhang Jiao <zhangjiao@cmss.chinamobile.com>

Merge pull request #30441 from anthonyeleven/master

osd: Change osd op queue cut off default to high

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30581 from liewegas/wip-write-returns

osd: allow rados write ops to return data and error codes

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #30454 from trociny/wip-41891

osd: fix possible crash on sending dynamic perf stats report

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30544 from tchaikov/wip-check-getentropy-at-runtime

auth/Crypto: fallback to /dev/urandom if getentropy() fails

Reviewed-by: Chang Liu <liuchang0812@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30526 from uweigand/endian-private

include,os: Make ceph_le member private

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #30525 into master

* refs/pull/30525/head:
qa/tasks/ceph.conf.template: disable power-of-2 warning
qa/standalone/mon/health-mute: use power of 2 for pg_num
osd/OSDMap: remove remaining g_conf() usage
PendingReleaseNotes: add note for 14.2.5 so we can backport this
osd/OSDMap: health alert for non-power-of-two pg_num

Reviewed-by: Kai Wagner <kwagner@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #30431 into master

* refs/pull/30431/head:
pybind/ceph_argparse: add :int or :float to numerical args
pybind/ceph_argparse: simplify osd name and target types
pybind/ceph_argparse: prefer field names to types in help output
pybind/ceph_argparse: more concise n=N '...'
pybind/ceph_argparse: [] (not {}) around optional args

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/dashboard: Support minimum password complexity rules (#29532)

mgr/dashboard: Support minimum password complexity rules

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: CephFS snapshots/quota REST API (#30256)

mgr/dashboard: CephFS snapshots/quota REST API

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge PR #30577 into master

* refs/pull/30577/head:
qa/suites/rados/thrash-old-clients: tolerate MON_DOWN

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #30599 from votdev/issue_41797_url_prefix

mgr/dashboard: Configuring an URL prefix does not work as expected

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #30597 from smithfarm/wip-opensuse-15.1

qa/distros/all: add openSUSE 15.1, drop openSUSE 12.2

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

Merge pull request #30430 from smithfarm/wip-release-schedule-14.2.4

doc/releases/schedule.rst: add 14.2.3, 14.2.4, 15.0.0 and drop dumpling

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #30478 from votdev/issue_41930_standby_dashboard

mgr/dashboard: Standby Dashboards don't handle all requests properly

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #30501 from liewegas/bugfix-41913

ceph-objectstore-tool: better error message if pgid and object do not match

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #30518 from liewegas/wip-mon-less-snap-metadata

mon/OSDMonitor: clean up removed_snap keys

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30530 from dzafman/wip-42015

osd: Remove unused osdmap flags full, nearfull from output

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #30406 into master

* refs/pull/30406/head:
doc: move MDS deployment from main quick start to CephFS quick start
doc: update CephFS Quick Start doc

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30561 into master

* refs/pull/30561/head:
doc/ceph-fuse: mention -k option in ceph-fuse man page

Reviewed-by: Douglas Fuller <dfuller@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge pull request #30404 from shun-s/cleanup-failure-pending

osd: make osd recover more smoothly by avoiding failure peer info to resent

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30517 from tchaikov/wip-test/dashboard/scheme-allow-unknown

qa/tasks/mgr/dashboard/test_health: add "allow_unknown" in mgr_map

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #30297 into master

* refs/pull/30297/head:
test_cephfs_shell: add note on how to run this testsuite
cephfs-shell: rewrite help text for put and get commands

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30533 from cxytt/fix-complete-to-not-reset

osd/PGLog: reset log.complete_to when recover obect failed

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30538 from tchaikov/wip/test/ceph_argparse/yet-another-test

mon.test: improve validation and add a test for "osd pool create"

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30504 from Songweibin/wip-app-meta-get

pybind/rados: add application_metadata_get

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30421 from sebastian-philipp/prometheus-always-on-modules

mgr/prometheus: Fix KeyError in get_mgr_status

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30200 from NancySu05/dev

mon/OSDMonitor: trim not-longer-exist failure reporters

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29906 into master

* refs/pull/29906/head:
vstart_runner: name booleans for options differently
qa/vstart_runner.py: add an option to remove old log
qa/vstart_runner.py: make log initialization code reusable
qa/vstart_runner.py: make printing of stdout of ps optional
qa/vstart_runner.py: add timeout for vstart.sh and stop.sh
qa/vstart_runner.py: add an option to teardown cluster

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #30612 from tchaikov/wip-install-deps.sh/do-not-fail-if-already-installed

install-deps: do not install if rpm already installed

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

install-deps: do not install if rpm already installed

there is chance that the system already have epel-release-latest-7.noarch.rpm
installed, in that case, install-deps.sh should not fail.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30466 from xiexingguo/wip-41924

osd/PGLog: persist num_objects_missing for replicas when peering is done

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30576 from xiexingguo/wip-42052

osd/OSDMap: do not trust partially simplified pg_upmap_item

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30560 from xiexingguo/wip-balancer-upmap-cli

mgr/balancer: enable pg_upmap cli for future use

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #30223 into master

* refs/pull/30223/head:
msg/async: do not register lossy client connections
msg/async: allow anonymous client-side connections

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mgr/dashboard: REST API: OpenAPI docs require internet connect… (#30477)

mgr/dashboard: REST API: OpenAPI docs require internet connection

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30557 from smithfarm/wip-42023

install-deps.sh: add EPEL repo for non-x86_64 archs as well

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mgr/dashboard: Configuring an URL prefix does not work as expected

Fixes: https://tracker.ceph.com/issues/41797
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #30590 from jtlayton/wip-42032

ceph.spec.in: fix Cython package dependency for Fedora

Merge pull request #29460 from rhcs-dashboard/bucket-versioning

mgr/dashboard: enable/disable versioning on RGW bucket

Reviewed-by: Volker Theile <vtheile@suse.com>

qa/distros/all: add openSUSE 15.1, drop openSUSE 12.2

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/tasks/ceph.conf.template: disable power-of-2 warning

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/mon/health-mute: use power of 2 for pg_num

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30208 from ricardoasmarques/support-iscsi-controls-types

mgr/dashboard: Controls UI inputs based on "type"

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #28834 into master

* refs/pull/28834/head:
libcephfs: Add test for lazyio via libcephfs
libcephfs: Add lazyio_propogate and lazyio_synchronize methods to libcephfs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #29818 into master

* refs/pull/29818/head:
client/MetaRequest: Add age to MetaRequest dump
osdc/Objecter: Add age to the ops
common/ceph_time: Use fixed floating-point notation for mono_clock

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29902 into master

* refs/pull/29902/head:
mds: Revert "properly setup client_need_snapflush for snap inode"
mds: cleanup dirty snap caps tracking

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29921 into master

* refs/pull/29921/head:
mds: split the dir if the op makes it oversized

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #30202 into master

* refs/pull/30202/head:
mds: Explicitly call slave_updates with 0 size
mds: Move log_segment_seq_t into class LogSegment
mds: Reorganize class members in LogSegment header

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30403 into master

* refs/pull/30403/head:
mds: mds returns -5 error when the deleted file does not exist

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #30462 into master

* refs/pull/30462/head:
qa: have kclient tests use new mount.ceph functionality

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

mgr/dashboard: Support iSCSI target-level CHAP auth (#30011)

mgr/dashboard: Support iSCSI target-level CHAP auth

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #30588 from ricardoasmarques/fix-iscsi-test-put

mgr/dashboard: Wait for iSCSI target put and delete

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: fix Cython package dependency for Fedora

Fedora distros do not have python3?-Cython packages, but they do have
python3-Cython ones. Fix the BuildRequires so that we only use the
python3_version_nodots based version string for RHEL.

Fixes: https://tracker.ceph.com/issues/42032
Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/dashboard: Wait for iSCSI target put and delete

Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #30571 from smithfarm/wip-bci-force

src/script/backport-create-issue: implement --force option

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Controls UI inputs based on "type"

iSCSI controls UI inputs should be rendered based
on the control "type" returned by `ceph-iscsi`.

Fixes: https://tracker.ceph.com/issues/41682
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: test_mgr_module QA test has not been adapted to… (#30513)

mgr/dashboard: Adapt test_mgr_module QA test to latest controller changes

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: wait for iscsi/target create/put (#30582)

mgr/dashboard: wait for iscsi/target create/put

Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: enable/disable versioning on RGW bucket

Fixes: https://tracker.ceph.com/issues/40920
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #30578 from tchaikov/wip-seastar-unix-domain-socket

seastar: pickup unix domain socket support

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

mgr/dashboard: wait for iscsi/target create/put

IscsiTarget.create() returns a Task, so we need to wait for its
completion before moving on.

the same applies to IscsiTarget.put()

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages/MOSDOpReply: ensure input payload is not passed via reply

The reply shouldn't include the input payload, for the sake of efficiency.

This has the slightly annoying side-effect that the class name and method
are no longer visible in osd_op_reply messages.  However, it does mean
that any data payload is not sent back to the client.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: make proxied writes behave with RETURNVEC

Signed-off-by: Sage Weil <sage@redhat.com>

cls/hello: test large return buffer overflow

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: impose a per-op return buffer limit (32)

Limit the return value size for writes, so that we don't inadvertantly
stuff too much data in the PGLog.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: include op_returns in pg log [dup] records, reply

If the op has the RETURNVEC flag set,

- Record the per-op return value and overall return code in the pg log and
  dup records.
- Include the same in the actual MOSDOpReply back to the client.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: reset payload_len in merge_osd_op_vector_out_data

This is cleaner and simpler.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: show in and out byte counts for OSDOp

This will appear in the MOSDOp[Reply] message strings.

Signed-off-by: Sage Weil <sage@redhat.com>

cls/hello,ceph_test_cls_hello: test returning data from a mutation

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: allow mutate ops to have return data

Signed-off-by: Sage Weil <sage@redhat.com>

librados: add a RETURNVEC flag

This signals that the client is soliciting and expecting return codes
that are >0 and/or return buffers on update ops.

This should be used with caution, since it means that compound requests
that include read and write requests will *also* persist the read ops in
order to be idempotent, which means that those ops' results may get
truncated.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add per-op return fields to log [dup] records

Allow an overall positive return value, and also per-op return value and
outdata for each op in a request.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: remove finish_proxy_write reply path

There is no case where we have already allocated a reply message here,
since that is only done in execute_ctx() after the transaction is prepared,
something that doesn't happen if we proxy the write through to the base
layer.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: drop unused MOSDOpReply creation path

It is unclear why this path was originally needed, but it does predate
the removal of the dual-ack removal, so I expect it was related to
that.  It is no longer necesarry.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: do not register lossy client connections

If our policy is server + lossy, we do not need to track our incoming
client connections by address.  First, because it doesn't do us any good.
Second, it is nicer if we don't, because we can allow multiple incoming
connections from the same peer addr.

Update a couple of tests: one doesn't apply any more, and the other needs
a different way of getting the just-accepted con ref.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: allow anonymous client-side connections

If the connection mode is lossy, allow us to open a new connection to
a target, regardless of whether other such connections are already open.
This allows for single-use connections.  If you call this multiple times,
you'll get separate, distinct connections.

We are lucky that the cleanup infrastructure for AsyncMessenger just works
without modification.  :)

Signed-off-by: Sage Weil <sage@redhat.com>

seastar: pickup unix domain socket support

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #30523 into master

* refs/pull/30523/head:
mon/MonClient: skip CEPHX_V2 challenge if client doesn't support it

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

osd/OSDMap: remove remaining g_conf() usage

Use the explicitly-passed cct here instead of relying on the global symbol.
The only user was check_health(), so this is easy.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: do not trust partially simplified pg_upmap_item

If we simplified a partially no-op pg_upmap_item, we shall still
continue to verify that the remaining part is valid.
The bug is introduced by 02e5499b350bcd7d9eac98b2072052a9a4a1f535,
before which we always validate the correctness of a pg_upmap_item
before trying to cancel or simplify it.

Fixes: https://tracker.ceph.com/issues/42052
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/suites/rados/thrash-old-clients: tolerate MON_DOWN

The first thing this test does is upgrade all mons.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Support iSCSI target-level CHAP auth

Requires `ceph-iscsi` config v11

Fixes: https://tracker.ceph.com/issues/41576
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Display the "destroyed" state in OSD list (#30514)

mgr/dashboard: Display the "destroyed" state in OSD list

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

install-deps.sh: add EPEL repo for non-x86_64 archs as well

Fixes: https://tracker.ceph.com/issues/42023
Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

src/script/backport-create-issue: implement --force option

If --force option is given along with an explicit list of issue numbers,
backport issues will be created regardless of issue status.

In all other usage scenarios, script behavior remains the same (backport
issues are created only if issue status is "Pending Backport").

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30420 from runsisi/wip-remove-requests

rpm,deb: python-requests is not needed for ceph-common

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: move MDS deployment from main quick start to CephFS quick start

Signed-off-by: Rishabh Dave <ridave@redhat.com>

doc: update CephFS Quick Start doc

Also, skip the details about CephX user's keyring and monitor's socket
since the kernel driver can figure out these details automatically now.

Fixes: https://tracker.ceph.com/issues/41872
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #30422 from rishabh-d-dave/cv-fix-pytest-warning

ceph-volume: fix warnings raised by pytest

doc/ceph-fuse: mention -k option in ceph-fuse man page

Fixes: https://tracker.ceph.com/issues/42044
Signed-off-by: Rishabh Dave <ridave@redhat.com>

vstart_runner: name booleans for options differently

This achieves 2 things: makes it more understandable for the reader that
the boolean variable is an option and reads less like a method name.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/vstart_runner.py: add an option to remove old log

Add an option "--clear-old-log" to get rid of log entries before making
entries for current batch of tests.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/vstart_runner.py: make log initialization code reusable

Move it to a method so that the code be can be reused.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/vstart_runner.py: make printing of stdout of ps optional

stdout of ps command is generally pretty huge which makes it harder to
interpret logs. Don't print it by default and add "--log-ps-output" to
enable printing it.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/vstart_runner.py: add timeout for vstart.sh and stop.sh

Override default timeout for execution of vstart.sh and stop.sh since
neither scripts take more than 100 seconds normally.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/vstart_runner.py: add an option to teardown cluster

Add "--teardown" option so that vstart_runner.py tears down Ceph
cluster automatically after the batch of tests has completed running.
Also delete build/dev and build/out directories, after running stop.sh.

Update the doc for vstart_runner.py as well.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

mgr/balancer: enable pg_upmap cli for future use

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PrimaryLogPG: remove unnecessary claim_op_out_data() calls

In all of these cases, the vector<OSDOp>* ops in question points to
the MOSDOp::ops vector, which is copied by the MOSDOpReply ctor.

Move back to a const MOSDOp* where appropropriate.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: do_pg_ops make MOSDOp const, fix comment

Signed-off-by: Sage Weil <sage@redhat.com>