osd/PrimaryLogPG: remove unused reply creation path

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: include op_returns in dup replies

We are storing the return metadata; actually use it when sending dup
replies!

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLog: drop unused reply_ctx() variant

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: remove dead already_ack()

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30617 from xiexingguo/wip-weird-daemon-key

mgr: fix weird health-alert daemon key

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: fix weird health-alert daemon key

Was:
  19 slow ops, oldest one blocked for 34 sec, daemons [osd,2,osd,4] have slow ops.

Now:
  153 slow ops, oldest one blocked for 38 sec, daemons [osd.3,osd.4,osd.5] have slow ops.

Fixes: https://tracker.ceph.com/issues/42079
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #29236 from liewegas/wip-read-hole-bypg

osd: implement per-pg leases to avoid stale reads

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #30527 from uweigand/bloom-filter-endian

common/bloom_filter: Fix endian issues

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30616 from ifed01/wip-ifed-fix-fsck

os/bluestore: fix invalid stray shared blob detection in fsck.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30625 from tchaikov/wip-42087

mgr/BaseMgrStandbyModule: drop GIL in ceph_get_module_option()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

crimson/osd: fix osdpg build

These are stubs; the reschule one (at minimum) probably needs a meaningful
implementation in order for the PG to peer in some cases.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: document new 'laggy' and 'wait' pg states

Signed-off-by: Sage Weil <sage@redhat.com>

doc: stale reads notes

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: skip wait state if osd set is empty

If there are no down OSDs from prior intervals, then the normal peering
process will end up contacting all of the prior OSDs and ensuring that
their prior interval is terminated during peering.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: make proc_lease, recalc_readable_until more verbose

Signed-off-by: Sage Weil <sage@redhat.com>

osd: is_replica() -> is_nonprimary()

The 'replica' term does not map well onto EC pools.  More importantly,
the implementation is often wrong for EC pools, where role may be 0 or 1
for EC pools independent of whether the OSD is the primary or not.

Introduce 'nonprimary' to mean an acting osd that is not the primary.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: piggyback pg_lease on MOSDPGLog

We only do this for primary -> replica, so we only need to proc_lease()
from the replica states.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: piggyback lease and ack on activation messages

The lease goes out with the MOSDPGLog or info, and the ack comes back with
the info.

We no longer need to renew the lease explicitly in
all_activated_and_committed() because we *just* piggybacked on activation.
We can just wait for the normal renew event to fire.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: renew before activate messages; send after activated

We want to renew before we prepeare or send activate messages so that we
have the opportunity to include leases in them (coming soon!).

And we do not want to send explicit lease messages until we know that the
peers have activate.  In particular, we want to avoid queueing a notify
(via pending_activators) and then sending a lease that will arrive before
it.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: notice 'dead' prior_readable OSDs

If we see that a prior_readable_down_osd is known to be dead, we can
remove it from the set.  And if the set is empty, we can skip the rest of
our waiting period and leave the WAIT state.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: track down OSDs relevant to prior_readable_until_ub

Keep track of which OSDs from the prior set we care about that affect
the prior_readable_until_ub.  Note that it is only the *down* OSDs that
we have to track here, since everything in the *probe* set we will already
contact during peering (they are still up), guaranteeing that those PGs
are aware of the interval change and are no longer readable in the prior
interval.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton-nomsg/osd_stale_reads.yaml

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30441 from anthonyeleven/master

osd: Change osd op queue cut off default to high

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30581 from liewegas/wip-write-returns

osd: allow rados write ops to return data and error codes

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #30454 from trociny/wip-41891

osd: fix possible crash on sending dynamic perf stats report

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30544 from tchaikov/wip-check-getentropy-at-runtime

auth/Crypto: fallback to /dev/urandom if getentropy() fails

Reviewed-by: Chang Liu <liuchang0812@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/BaseMgrStandbyModule: drop GIL in ceph_get_module_option()

always drop GIL when performing block ops

Fixes: https://tracker.ceph.com/issues/42087
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30526 from uweigand/endian-private

include,os: Make ceph_le member private

Reviewed-by: Kefu Chai <kchai@redhat.com>

bloom-filter: Remove POD overloads for insert and contains

These are not used in Ceph code currently, and should not
be used in the future either, since any use will introduce
byte-order dependent behavior.  Remove them to prevent
accidental use.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

bloom-filter: Improve test cases

The integer bloom filter test cases do not really match typical usage
of the bloom filter in actual Ceph code.  In particular:

- the tests use consecutive integer ranges, while Ceph code uses
  hash values uniformly distributed over the uint32_t space;

- the tests pass "int" to the insert and contains functions, which
  causes the generic C++ POD type overload to be selected instead
  of the uint32_t overload that is used by Ceph code.  The POD
  overload is dependent on host byte order, and behaves actually
  different from the uint32_t overload on little-endian systems.

To fix these issues, this patch changes the integer tests to
always pass in uint32_t (instead of int), and to use results
of a pseudo-random number generator instead of consecutive
sequences.  (We assume the period of the generator is long
enough that all values generated within one test instance
are distinct.)

This not only makes the test pass on both big- and little-endian
hosts now, but it also allows tightening of the allowable actual
false positive rates, as they now match much closer the expected
values.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

Merge PR #30525 into master

* refs/pull/30525/head:
qa/tasks/ceph.conf.template: disable power-of-2 warning
qa/standalone/mon/health-mute: use power of 2 for pg_num
osd/OSDMap: remove remaining g_conf() usage
PendingReleaseNotes: add note for 14.2.5 so we can backport this
osd/OSDMap: health alert for non-power-of-two pg_num

Reviewed-by: Kai Wagner <kwagner@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #30431 into master

* refs/pull/30431/head:
pybind/ceph_argparse: add :int or :float to numerical args
pybind/ceph_argparse: simplify osd name and target types
pybind/ceph_argparse: prefer field names to types in help output
pybind/ceph_argparse: more concise n=N '...'
pybind/ceph_argparse: [] (not {}) around optional args

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/dashboard: Support minimum password complexity rules (#29532)

mgr/dashboard: Support minimum password complexity rules

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: CephFS snapshots/quota REST API (#30256)

mgr/dashboard: CephFS snapshots/quota REST API

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge PR #30577 into master

* refs/pull/30577/head:
qa/suites/rados/thrash-old-clients: tolerate MON_DOWN

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #30599 from votdev/issue_41797_url_prefix

mgr/dashboard: Configuring an URL prefix does not work as expected

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

os/bluestore: fix invalid stray shared blob detection in fsck.

This is a regression, caused by https://github.com/ceph/ceph/pull/30264

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #30597 from smithfarm/wip-opensuse-15.1

qa/distros/all: add openSUSE 15.1, drop openSUSE 12.2

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

Merge pull request #30430 from smithfarm/wip-release-schedule-14.2.4

doc/releases/schedule.rst: add 14.2.3, 14.2.4, 15.0.0 and drop dumpling

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #30478 from votdev/issue_41930_standby_dashboard

mgr/dashboard: Standby Dashboards don't handle all requests properly

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #30501 from liewegas/bugfix-41913

ceph-objectstore-tool: better error message if pgid and object do not match

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #30518 from liewegas/wip-mon-less-snap-metadata

mon/OSDMonitor: clean up removed_snap keys

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30530 from dzafman/wip-42015

osd: Remove unused osdmap flags full, nearfull from output

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #30406 into master

* refs/pull/30406/head:
doc: move MDS deployment from main quick start to CephFS quick start
doc: update CephFS Quick Start doc

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30561 into master

* refs/pull/30561/head:
doc/ceph-fuse: mention -k option in ceph-fuse man page

Reviewed-by: Douglas Fuller <dfuller@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge pull request #30404 from shun-s/cleanup-failure-pending

osd: make osd recover more smoothly by avoiding failure peer info to resent

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30517 from tchaikov/wip-test/dashboard/scheme-allow-unknown

qa/tasks/mgr/dashboard/test_health: add "allow_unknown" in mgr_map

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #30297 into master

* refs/pull/30297/head:
test_cephfs_shell: add note on how to run this testsuite
cephfs-shell: rewrite help text for put and get commands

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30533 from cxytt/fix-complete-to-not-reset

osd/PGLog: reset log.complete_to when recover obect failed

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30538 from tchaikov/wip/test/ceph_argparse/yet-another-test

mon.test: improve validation and add a test for "osd pool create"

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30504 from Songweibin/wip-app-meta-get

pybind/rados: add application_metadata_get

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30421 from sebastian-philipp/prometheus-always-on-modules

mgr/prometheus: Fix KeyError in get_mgr_status

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30200 from NancySu05/dev

mon/OSDMonitor: trim not-longer-exist failure reporters

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29906 into master

* refs/pull/29906/head:
vstart_runner: name booleans for options differently
qa/vstart_runner.py: add an option to remove old log
qa/vstart_runner.py: make log initialization code reusable
qa/vstart_runner.py: make printing of stdout of ps optional
qa/vstart_runner.py: add timeout for vstart.sh and stop.sh
qa/vstart_runner.py: add an option to teardown cluster

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #30612 from tchaikov/wip-install-deps.sh/do-not-fail-if-already-installed

install-deps: do not install if rpm already installed

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

install-deps: do not install if rpm already installed

there is chance that the system already have epel-release-latest-7.noarch.rpm
installed, in that case, install-deps.sh should not fail.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30466 from xiexingguo/wip-41924

osd/PGLog: persist num_objects_missing for replicas when peering is done

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30576 from xiexingguo/wip-42052

osd/OSDMap: do not trust partially simplified pg_upmap_item

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30560 from xiexingguo/wip-balancer-upmap-cli

mgr/balancer: enable pg_upmap cli for future use

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #30223 into master

* refs/pull/30223/head:
msg/async: do not register lossy client connections
msg/async: allow anonymous client-side connections

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

osd: send ops back to primary if replica is not readable

This is the simplest strategy--much simpler than queueing them and
waking them up again later.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: set WAIT state and block ops to wait for prior readable_until

If we start a new interval and the prior interval may have OSDs that
are still readable, set the WAIT state bit and block operations until
sufficient time has elapsed.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add CheckReadable pg event, queue_check_readable()

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: update prior_readable_until_ub on new interval

If a new interval starts, apply the old interval's readable_until_ub to
our prior.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: refresh prior_readable_until_ub in pg_history_t on share

Before we share pg_history_t, refresh the prior_readable_until_ub to be
a simple duration from *now*, so that it is completely clock-independent.
The receiver can interpret it based on the receive time for the message,
which loses a bit of precision but is safe since this is an upper bound.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: requeue ops when PG is no longer laggy

When we get a lease_ack, leave the laggy state.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: put ops on waitlist when PG is laggy

Move PG to laggy state if we aren't readable

Signed-off-by: Sage Weil <sage@redhat.com>

crimson/osd/pg: add schedule_renew_lease stub

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: do full on_new_interval on new peering interval

This looks like a bug from the PeeringState transition.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: schedule regular lease renewals

Signed-off-by: Sage Weil <sage@redhat.com>

osd: send and process lease[_ack] messages

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add lease/readable tracking members, helpers

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add pg_lease[_ack]_t

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add mono_timer

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add READ_LEASE_INTERVAL pool property, osd_pool_readable_min_ratio

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringStage: PGPool::get_readable_interval() accessor

For now, just do the heartbeat interval * 2.  Eventually we'll want a
pool property.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add PG_STATE_WAIT

PG is waiting for previous intervals' readable intervals to expire.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add PG_STATE_LAGGY

PG is laggy (unreadable) because ping(s) are delayed.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: pg_history_t: add prior_readable_until_ub

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: REST API: OpenAPI docs require internet connect… (#30477)

mgr/dashboard: REST API: OpenAPI docs require internet connection

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30557 from smithfarm/wip-42023

install-deps.sh: add EPEL repo for non-x86_64 archs as well

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

osd/PeeringState: PG state 64 bits wide

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Configuring an URL prefix does not work as expected

Fixes: https://tracker.ceph.com/issues/41797
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #30590 from jtlayton/wip-42032

ceph.spec.in: fix Cython package dependency for Fedora

Merge pull request #29460 from rhcs-dashboard/bucket-versioning

mgr/dashboard: enable/disable versioning on RGW bucket

Reviewed-by: Volker Theile <vtheile@suse.com>

qa/distros/all: add openSUSE 15.1, drop openSUSE 12.2

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/tasks/ceph.conf.template: disable power-of-2 warning

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/mon/health-mute: use power of 2 for pg_num

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30208 from ricardoasmarques/support-iscsi-controls-types

mgr/dashboard: Controls UI inputs based on "type"

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #28834 into master

* refs/pull/28834/head:
libcephfs: Add test for lazyio via libcephfs
libcephfs: Add lazyio_propogate and lazyio_synchronize methods to libcephfs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #29818 into master

* refs/pull/29818/head:
client/MetaRequest: Add age to MetaRequest dump
osdc/Objecter: Add age to the ops
common/ceph_time: Use fixed floating-point notation for mono_clock

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29902 into master

* refs/pull/29902/head:
mds: Revert "properly setup client_need_snapflush for snap inode"
mds: cleanup dirty snap caps tracking

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29921 into master

* refs/pull/29921/head:
mds: split the dir if the op makes it oversized

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #30202 into master

* refs/pull/30202/head:
mds: Explicitly call slave_updates with 0 size
mds: Move log_segment_seq_t into class LogSegment
mds: Reorganize class members in LogSegment header

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30403 into master

* refs/pull/30403/head:
mds: mds returns -5 error when the deleted file does not exist

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #30462 into master

* refs/pull/30462/head:
qa: have kclient tests use new mount.ceph functionality

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

mgr/dashboard: Support iSCSI target-level CHAP auth (#30011)

mgr/dashboard: Support iSCSI target-level CHAP auth

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #30588 from ricardoasmarques/fix-iscsi-test-put

mgr/dashboard: Wait for iSCSI target put and delete

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: fix Cython package dependency for Fedora

Fedora distros do not have python3?-Cython packages, but they do have
python3-Cython ones. Fix the BuildRequires so that we only use the
python3_version_nodots based version string for RHEL.

Fixes: https://tracker.ceph.com/issues/42032
Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/dashboard: Wait for iSCSI target put and delete

Signed-off-by: Ricardo Marques <rimarques@suse.com>