doc: split up SubmittingPatches.rst

Split the existing SubmittingPatches.rst into three files:

* SubmittingPatches.rst (general guidelines, for GitHub/master branch)
* SubmittingPatches-kernel.rst (for kernel patches)
* SubmittingPatches-backports.rst (for backports)

Fixes: http://tracker.ceph.com/issues/20953
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30654 from ajarr/fix-42096

mgr/volumes: fix incorrect snapshot path creation

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #30627 into master

* refs/pull/30627/head:
qa/suites/rados/verify: whitelist MON_DOWN when using valgrind
qa/suits/rados/basic/tasks/rados_api_tests: pgs can go degraded

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #30640 into master

* refs/pull/30640/head:
osd/PrimaryLogPG: remove unused reply creation path
osd/PrimaryLogPG: include op_returns in dup replies
osd/PrimaryLog: drop unused reply_ctx() variant
osd/PrimaryLogPG: remove dead already_ack()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #30596 into master

* refs/pull/30596/head:
qa/suites/rados/singleton/all/ec-lost-unfound: no rbd pool

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30675 from idryomov/wip-set-kernel-secret

common/secret.c: don't pass uninitialized stack data to the kernel

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Update npm packages (#30439)

mgr/dashboard: Update npm packages

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #30626 into master

* refs/pull/30626/head:
doc: document recover_session= option in mount.ceph manpage
doc: update documents on using kcephfs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30542 into master

* refs/pull/30542/head:
doc: cleanup CephFS Landing Page

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Show smartctl summary in OSD tab (#30378)

mgr/dashboard: Show smartctl summary in OSD tab

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29922 from smithfarm/wip-41524

cmake: set empty RPATH for some test executables

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #30416 into master

* refs/pull/30416/head:
client: add procession of SEEK_HOLE and SEEK_DATA in lseek.

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30653 from tchaikov/wip-os/drop-kinetic

os: remove KineticStore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28569 from alimaredia/wip-rgw-ioctx-operate-replace

rgw: replace direct calls to ioctx.operate()

doc: document recover_session= option in mount.ceph manpage

Fixes: https://tracker.ceph.com/issues/40957
Signed-off-by: Jeff Layton <jlayton@redhat.com>

common/secret.c: don't pass uninitialized stack data to the kernel

ceph_unarmor() returns the number of bytes decoded, which can be (and
usually is) smaller than the size of the payload array.

set_kernel_secret() has behaved this way ever since it was introduced
in commit bee85518e288 ("mount.ceph: Use kernel key management API when
possible.").  The reason it didn't cause problems in the kernel is that
the encoding includes the actual length of the secret and there is no
check for the end of the supplied payload (see ceph_key_preparse() in
net/ceph/crypto.c).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #30655 from tchaikov/wip-cmake/crimson/pass-args

cmake: pass arguments to crimson tests

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

client: add procession of SEEK_HOLE and SEEK_DATA in lseek.

Fixes: https://tracker.ceph.com/issues/42107
Signed-off-by: Shen Hang <harryshen18@gmail.com>

Merge pull request #30613 from joke-lee/rgw-ceph-dencoder-add-RGWPeriodLatestEpochInfo

ceph-dencoder: add RGWPeriodLatestEpochInfo support.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Update npm packages

Signed-off-by: Tiago Melo <tmelo@suse.com>

doc: update documents on using kcephfs

Overhaul the fstab.rst and kernel.rst files with more relevant examples
now that the mount helper does autodiscovery. Add a preamble to
kernel-features.rst and add a warning about the deprecation of inline
data.

Fixes: https://tracker.ceph.com/issues/42025
Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/volumes: fix incorrect snapshot path creation

When constructing the snapshot path, the components are a mix of
byte and string type objects. The 'os.path.join()' method needs
the components to be of the same type. Hence convert all components
to byte type objects.

Introduced-by: 3d63cd947f55a8508f972ee249ea6a04836429ae
Fixes: https://tracker.ceph.com/issues/42096
Signed-off-by: Ramana Raja <rraja@redhat.com>

cmake: pass arguments to crimson tests

otherwise they will take all CPU cores and all memory

Signed-off-by: Kefu Chai <kchai@redhat.com>

build/ops: set empty RPATH for some test executables

Fixes: https://tracker.ceph.com/issues/41524
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30587 from sebastian-philipp/python-common-readme

python-common: Add small Readme

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge PR #30341 into master

* refs/pull/30341/head:
cephfs-shell: better complain info, when deleting non-empty directory

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge PR #30559 into master

* refs/pull/30559/head:
mds: Reorganize class members in MDBalancer header

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #30580 into master

* refs/pull/30580/head:
client: don't ceph_abort on bad llseek whence value
client: remove Inode dir_contacts field

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30407 from joscollin/wip-B41841-yes-really-mean-it

mgr/volumes: protection for `fs volume rm` command

doc: cleanup CephFS Landing Page

Fixes: https://tracker.ceph.com/issues/41952
Signed-off-by: Milind Changire <mchangir@redhat.com>

os: remove KineticStore

kinetic-cpp-client and kinetic project as a whole are not acitvely
supported or maintained anymore. see

- https://github.com/Kinetic/kinetic-cpp-client
- http://www.openkinetic.org

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #30552 into master

* refs/pull/30552/head:
cmake: fix libtsan detection
test: librados startup/shutdown racer test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #30630 from tchaikov/wip-build-on-centos8

install-deps,rpm,do_cmake: build on RHEL/CentOS 8

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #30531 from alimaredia/wip-rgw-maven-version-update

qa: bump maven repo version in s3a_hadoop.py

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30615 from xiexingguo/wip-progress-time-remaining

mgr/progress: estimated remaining time for events

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30591 from xiexingguo/wip-balancer-throttler

mgr/balancer: upmap_max_iterations -> upmap_max_optimizations; behave as it is per pool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29674 from athanatos/sjust/wip-fio-trace

os/bluestore: expand lttng tracepoints, improve fio_ceph_objectstore backend

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cls: CLS_CLIET_HIDE_IOCTX changes in cls headers

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: various ioctx.operate refactoring changes

- remove unused #include for cls_client headers
- refactor cls_timeindex_trim call in
rgw_object_expirer_core.cc
- add more CLS_CLIENT_HIDE_IOCTX in header files
- move target_compile_definitions to rgw_a target
- make repeat cls helper functions static

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: overload cls_rgw functions that do io_ctx.operate()

Call the new overloaded functions and call
rgw_rados_operate after them.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cls: remove unused cls_rgw_reshard_get_head()

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: replace ioctx.operate() with rgw_rados_operate()

For all operate calls in:
rgw_op.cc
rgw_orphan.cc
rgw_reshard.cc
rgw_rados.cc

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: fix libtsan detection

This variable is case-sensitive.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

osd/PrimaryLogPG: remove unused reply creation path

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: include op_returns in dup replies

We are storing the return metadata; actually use it when sending dup
replies!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30620 from votdev/issue_42069_validate_bucket_name

mgr/dashboard: Bucket names cannot be formatted as IP address

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kai Wagner <kwagner@suse.com>

Merge pull request #30647 from s0nea/wip-dashboard-fix-typo

mgr/dashboard: fix small typos in description message

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29925 from sebastian-philipp/run_tox-overwrite-build-dir

script/run_tox.sh: Don't overwrite the build dir

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix two small typos

Remove unnecessary blanks.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

script/run_tox.sh: Don't overwrite the Ceph's bulid dir

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #30639 from tchaikov/wip-crimson-readable

crimson/osd: implement readable/lease related methods

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30644 from majianpeng/osd-remove-unused-func

osd: remove unused function

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30632 from xiexingguo/wip-kick-recovery-pri

osd/PrimaryLogPG: always use strict priority ordering for kicked recovery ops

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30642 from Rjerk/wip-fix-doc

doc/radosgw: fix typos

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/PrimaryLogPG: add closing ")" in log

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/pg: implement schedule_renew_lease()

see also d883db70282c406edaceb72a90ecddccd97a0161

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: implement ShardServices::get_hb_stamps()

PeeringState::proc_lease() requires a valid hb_stamps[0], which is in
turned maintained by `pl`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/pg: implement queue_check_readable() and recheck_readable()

see also 7aec060e0aff4588ff51f744dd194d1c0f7793c2

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/ceph_time: do not handle negative case

this change partially reverts 353a0e5f, unlike `signedspan`,
`timespan::rep` is unsigned. so no need to handle negative case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Bucket names cannot be formatted as IP address

In general, bucket names should follow domain name constraints:
- Bucket names must be unique.
- Bucket names cannot be formatted as IP address.
- Bucket names can be between 3 and 63 characters long.
- Bucket names must not contain uppercase characters or underscores.
- Bucket names must start with a lowercase letter or number.
- Bucket names must be a series of one or more labels. Adjacent labels are separated by a single period (.). Bucket names can contain lowercase letters, numbers, and hyphens. Each label must start and end with a lowercase letter or a number.

Fixes: https://tracker.ceph.com/issues/42069
Signed-off-by: Volker Theile <vtheile@suse.com>

osd: remove unused function

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge PR #30396 into master

* refs/pull/30396/head:
doc: document mds journaling

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30463 into master

* refs/pull/30463/head:
vstart_runner: allow the use of it with kernel mounts

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/radosgw: fix typo

Signed-off-by: Liu Lan <liulan@umcloud.com>

mgr/progress: estimated remaining time for events

Fixes: https://tracker.ceph.com/issues/40419
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30638 from uweigand/cmake-atomic-16byte

cmake: Test for 16-byte atomic support on IBM Z

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/PrimaryLog: drop unused reply_ctx() variant

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: remove dead already_ack()

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/verify: whitelist MON_DOWN when using valgrind

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: Test for 16-byte atomic support on IBM Z

On IBM Z the Boost tagged pointer implementation cannot use
"pointer compression" as there are no unused bits in an address;
the whole 64-bit address space is available to user space code.

Instead, Boost uses 16-byte atomics.  This is always supported
on IBM Z, but depending on the particular compiler (version)
it may require linking against libatomic.  The existing checks
in CheckCxxAtomic.cmake do not catch this, however, as they only
test for (up to) 8-byte atomic support.

Fixed by adding a test for 16-byte atomic support on IBM Z.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

Merge pull request #30617 from xiexingguo/wip-weird-daemon-key

mgr: fix weird health-alert daemon key

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: fix weird health-alert daemon key

Was:
  19 slow ops, oldest one blocked for 34 sec, daemons [osd,2,osd,4] have slow ops.

Now:
  153 slow ops, oldest one blocked for 38 sec, daemons [osd.3,osd.4,osd.5] have slow ops.

Fixes: https://tracker.ceph.com/issues/42079
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #29236 from liewegas/wip-read-hole-bypg

osd: implement per-pg leases to avoid stale reads

Reviewed-by: Samuel Just <sjust@redhat.com>

osd/PrimaryLogPG: always use strict priority ordering for kicked recovery ops

See 590c4979d40ea2073da8180140303ac067d87cc4 for the background of
this change.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30527 from uweigand/bloom-filter-endian

common/bloom_filter: Fix endian issues

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30616 from ifed01/wip-ifed-fix-fsck

os/bluestore: fix invalid stray shared blob detection in fsck.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30625 from tchaikov/wip-42087

mgr/BaseMgrStandbyModule: drop GIL in ceph_get_module_option()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Revert "osd: give recovery ops initialized by client op a higher priority"

This reverts commit c0f87e0f91539d4d12dad5bddf95a97457a21c67.

The 'osd_op_queue_cut_off' config option determines which level of
high priority ops should use strict priority ordering and may change
from time to time. Since the main strategy of 'osd_kick_recovery_op_priority'
is to simply follow up 'osd_op_queue_cut_off', we can instead make a direct
use of 'osd_op_queue_cut_off' to achieve the same thing explicitly.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph-dencoder: add RGWPeriodLatestEpochInfo support.

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

crimson/osd: fix osdpg build

These are stubs; the reschule one (at minimum) probably needs a meaningful
implementation in order for the PG to peer in some cases.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: document new 'laggy' and 'wait' pg states

Signed-off-by: Sage Weil <sage@redhat.com>

doc: stale reads notes

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: skip wait state if osd set is empty

If there are no down OSDs from prior intervals, then the normal peering
process will end up contacting all of the prior OSDs and ensuring that
their prior interval is terminated during peering.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: make proc_lease, recalc_readable_until more verbose

Signed-off-by: Sage Weil <sage@redhat.com>

osd: is_replica() -> is_nonprimary()

The 'replica' term does not map well onto EC pools.  More importantly,
the implementation is often wrong for EC pools, where role may be 0 or 1
for EC pools independent of whether the OSD is the primary or not.

Introduce 'nonprimary' to mean an acting osd that is not the primary.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: piggyback pg_lease on MOSDPGLog

We only do this for primary -> replica, so we only need to proc_lease()
from the replica states.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: piggyback lease and ack on activation messages

The lease goes out with the MOSDPGLog or info, and the ack comes back with
the info.

We no longer need to renew the lease explicitly in
all_activated_and_committed() because we *just* piggybacked on activation.
We can just wait for the normal renew event to fire.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: renew before activate messages; send after activated

We want to renew before we prepeare or send activate messages so that we
have the opportunity to include leases in them (coming soon!).

And we do not want to send explicit lease messages until we know that the
peers have activate.  In particular, we want to avoid queueing a notify
(via pending_activators) and then sending a lease that will arrive before
it.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: notice 'dead' prior_readable OSDs

If we see that a prior_readable_down_osd is known to be dead, we can
remove it from the set.  And if the set is empty, we can skip the rest of
our waiting period and leave the WAIT state.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: track down OSDs relevant to prior_readable_until_ub

Keep track of which OSDs from the prior set we care about that affect
the prior_readable_until_ub.  Note that it is only the *down* OSDs that
we have to track here, since everything in the *probe* set we will already
contact during peering (they are still up), guaranteeing that those PGs
are aware of the interval change and are no longer readable in the prior
interval.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton-nomsg/osd_stale_reads.yaml

Signed-off-by: Sage Weil <sage@redhat.com>

do_cmake.sh: disable amqp and rdma on RHEL/CentOS 8

RHEL/CentOS 8 does not offer librabbitmq-devel, libibverbs-devel and
librdmacm-devel at this moment. so disable these features now.

Signed-off-by: Kefu Chai <kchai@redhat.com>

do_cmake.sh: set WITH_RADOSGW_AMQP_ENDPOINT using ARGS

so we can extend ARGS in a more unified way

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: disable "make check" for rhel8

following packages are not available on RHEL/CentOS 8, they are for
dashboard unit tests. so disable all python packages for dashboard
tests for now:

python-cherrypy
python-coverage
python-pecan
python-routes
python-tox
xmlsec1-devel

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: disable cephfs_java and amqp_endpoint if RHEL8

RHEL/CentOS 8 does not offer junit and sharutils yet, so disable
cephfs_java on rhel8 at this moment.

RHEL/CentOS 8 does not offer librabbitmq-devel yet, so disable
amqp_endpoint on rhel8 at this moment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: only install python-srpm-macros for required macros

the reason why we need to install these macros is to solve the
egg-chicken problem -- to set `_python_buildid` and `python3_pkgversion`
so that we can prepare the build dependencies and install them. in which,
`_python_buildid` is defined using `python3_pkgversion`. this macro is
offered by python-srpm-macros.

the other macros, like `python3_sitelib` and `__python3` are offered by
`python3-rpm-macros`.

this change also avoid the issue if we install `*rpm-macros` on CentOS8:

Error:
 Problem: package R-rpm-macros-1.1.0-2.el8.noarch requires /usr/bin/Rscript, but none of the providers can be installed
  - package R-rpm-macros-1.1.0-2.el8.noarch requires R-core, but none of the providers can be installed
  - conflicting requests
  - nothing provides libRblas.so()(64bit) needed by R-core-3.6.1-1.el8.x86_64
  - nothing provides openblas-Rblas needed by R-core-3.6.1-1.el8.x86_64
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30441 from anthonyeleven/master

osd: Change osd op queue cut off default to high

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30581 from liewegas/wip-write-returns

osd: allow rados write ops to return data and error codes

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #30454 from trociny/wip-41891

osd: fix possible crash on sending dynamic perf stats report

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30544 from tchaikov/wip-check-getentropy-at-runtime

auth/Crypto: fallback to /dev/urandom if getentropy() fails

Reviewed-by: Chang Liu <liuchang0812@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>