osd/OSDMap: don't warn on NEW osd ids

If we allocate a new OSD, don't raise a health alert about it.

Signed-off-by: Sage Weil <sage@newdream.net>

mon/OSDMonitor: behave if inc map sets weight on not-yet-existing OSD

Signed-off-by: Sage Weil <sage@newdream.net>

mon/OSDMonitor: do not mark newly created OSDs OUT

Currently, new OSDs are marked OUT.

This behavior appears to date back all the way to the 'osd new' command
in c9e6cac1cfae73cc4bce4a995ed2bba5dc85ae4b, and for 'osd create' from
118f081b3c64bc36a8b9825e3373587a334c672b.  The first commit has no
real explanation, but it presumably inherited it from the second. That
second commit, though, says

    if we are creating an osd which has the same id as a previously
    removed 'in' osd, we should not mark this newly created osd as 'in'

This isn't actually a good idea, however.  If we are creating (or reusing)
a new OSD id, the OSD that starts up will have no data.  So no matter what
there will be a data migration from the before state to the final state.
If we mark the osd OUT when the osd id is allocated but before the OSD
starts up, we'll create a middle state where PGs are mapped to the id (by
virtue of the CRUSH weight) and then remapped away (due to out), creating
a middle state where a bunch of PGs will repeer and maybe data will move.

Instead, we have two cases:

1) If we are reusing a DESTROYED osd id, we should leave the in/out
state the way it was.  This way we still go straight from the before
state to the after state (the osd will mark itself in when it starts up).

2) If we are allocating a new id in do_osd_create(), we want the OSD
to be IN, so there is no middle state.  Unfortunately, we have to work
around apply_incremental() being obnoxious here: it's sloppy implementation
will implicitly set EXISTS by virtue of new_osd_weight (the mark IN part)
before applying the osd_state XOR, so be careful!  (This behavior is
mirrored by the Linux kernel implementation too, thankfully.)

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #39138 into master

* refs/pull/39138/head:
qa: valgrind test for cephfs-mirror daemon
cephfs-mirror: use preforker for daemonizing
test: adjust sleep time to account for valgrind runs
cephfs-mirror: gracefully shutdown threads, timers, etc..
cephfs-mirror: call ceph_release() to cleanup mount alloc
cephfs-mirror: shutdown filesystem/cluster connections on shutdown
cephfs-mirror: set init failed flag on FSMirror::init() failure

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39618 into master

* refs/pull/39618/head:
doc: fix broken links multimds and kcephfs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39620 from rzarzynski/wip-crimson-monc-failure-handling

crimson: multiple fixes for connection failure handling in MonClient

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39405 from rhcs-dashboard/cherrypy-security

mgr/dashboard: set security headers

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

crimson/monc: drop the pending connection on ms_handle_reset().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/monc: don't assume reopen_session() shall establish active_con.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc: fix broken links multimds and kcephfs

Fixes: https://tracker.ceph.com/issues/49372
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #39432 from dillaman/wip-49238

rbd-mirror: delay update snapshot mirror image state

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #39328 into master

* refs/pull/39328/head:
tools/rados: replace omap_key_valid with std::optional.
tools/rados: replace omap_key_pretty with prettify().
tools/rados: add support for binary object names.

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #39394 into master

* refs/pull/39394/head:
tools/kvstore-tool: implement 'dissect' command
kv/KeyValueHistogram: make Bluestore's DBHistogram class reusable.

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge PR #39460 into master

* refs/pull/39460/head:
msg/async/dpdk: be compatible with DPDK 19.11
common/Tub: remove Tub.h
cmake: use NUMA for package name in FindNUMA.cmake
msg/async/dpdk: define MARKER in .hh
msg/async/dpdk: use optional<> instead of Tub<>
compressor/lz4: use optional<> instead of Tub<>
compressor/lz4: use range-based loop

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #39495 into master

* refs/pull/39495/head:
mgr/cephadm: remove spec from CephadmDaemonDeploySpec

Reviewed-by: Adam King <adking@redhat.com>

Merge PR #39535 into master

* refs/pull/39535/head:
osd: Don't clear m_planned_scrub until scrub finishes successfully

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #39478 from orozery/rbd-encryption-doc

doc/rbd: rbd encryption

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #39504 into master

* refs/pull/39504/head:
mgr/cephadm/upgrade: restart mgr after mons upgrade to pacific
mgr: rename config_prefix -> mgr_store_prefix
mgr: fix config option prefix
mgr/MgrStandby: simplify weird code
mgr/cephadm: use get_foreign_ceph_option() instead of 'config get' mon command
mgr_module: get_foreign_ceph_option(entity, name)
mgr: maintain a ConfigMap copy
mon/ConfigMap: factor out parse_key helper
mon/ConfigMap: fix stray option leak
mgr: use new kv subscription for mgr/, device/, config/
mon: allow subscription to kv/config-key data
mon: convert ConfigKeyService -> KVMonitor

Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge pull request #39348 from xxhdx1985126/wip-crimson-client-request-sequencer

crimson/osd: client request sequencer

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39552 from smithfarm/wip-49356

rpm: openSUSE Tumbleweed has lua54 now

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: log when obc lock is acquired

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: add I/O sequencer to preserve client_requests' order across PG interval change

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: interrupt all client_requests that are waiting for acquiring obc locks

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

doc/rbd: rbd encryption

This commit adds documentation of the new rbd encryption feature

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #39586 from liewegas/bug-45441

qa/suites/rados/singletone: ignorelist MON_DOWN when injecting msgr errors

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #39587 from dillaman/wip-test-neorados

cmake: libneoradostest-support should be static

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39591 from tchaikov/wip-mgr-diskpred-ann

mgr/diskprediction_local: add typing annotations and cleanups

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm/upgrade: restart mgr after mons upgrade to pacific

If we are upgrading to pacific, we need to restart the mgr *after* the
mons have a pacific quorum so that they can get the kv subscriptions.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39539 from lixiaoy1/enable_pwl

cmake: enable write-back cache in spec

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39567 from lixiaoy1/fix_read_ios

librbd/cache/pwl: wait for the completion of reads

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39522 from trociny/wip-49284

mgr/rbd_support: mirror snapshot schedule should skip non-primary images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39451 from trociny/wip-49282

librbd: allow disabling journaling for snapshot based mirroring image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39481 from trociny/wip-luks-encryption-test-sudo

qa/workunits/rbd: make luks-encryption test work on vstart cluster

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #39582 into master

* refs/pull/39582/head:
mon: fix INCOMPAT_QUINCY ondisk compatset feature bit

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39590 from tchaikov/wip-49395

ceph.spec.in: s/%cmake/cmake/

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39592 from tchaikov/wip-crimson-read-exactly

crimson/net: throw read_eof if short read

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: throw read_eof if short read

as per the implementation and the document of input_stream::read_exactly():

/// stream and returns them. If the end of stream is reached before n
/// bytes were read, fewer than n bytes will be returned - so despite
/// the method's name, the caller must not assume the returned buffer
/// will always contain exactly n bytes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/diskprediction_local: add typing annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/diskprediction_local: raise if Predictor cannot be initialized

simpler this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/diskprediction_local: add Predictor abstract class

to make the interface more explicit, also pave the road to mypy.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/diskprediction_local: drop unused COMMANDS

the base class of `MgrModule` already has it

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/diskprediction_local: use "int" for interval options

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/diskprediction_local: define options using Option

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/diskprediction_local: add to flake8

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39588 from athanatos/sjust/wip-fix-store-nbd

crimson/tools/store-nbd: fix invalid buffer access on connection termination

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: s/%cmake/cmake/

this change partially reverts da7030db79c5dbd480040eb249a76638caf52707
which use %cmake rpm macro in the place of "cmake". but

%cmake sets BUILD_SHARED_LIBS=ON. so quite a few internal libraries
defined using add_library() are now compiled into shared libraries which
are not installed or packagesd. when we are installing the rpm packages
compiled with this option, rpm compiles because the linked libraries are
missing, for instance, `libgmock.so.1.10.0` was compiled as a static
library before da7030db79c5dbd480040eb249a76638caf52707, and was
included by the test executables. but after that change it's compiled
as a shared library.

so we need to either package the linked shared libraries or just link
against them statically. at this moment, the latter approach is simpler,
albeit larger size of exectuable and dbg symbols.

Fixes: https://tracker.ceph.com/issues/49395
Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store-nbd: actually check magic

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #39573 from weixinwei/master

librbd: memory is not preallocated in rbd c++ read api

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/tools/store-nbd: check length of buffer

read_exactly does not return an error if the stream
has ended.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #39585 from ivancich/wip-clean-rgw-warnings-1

rgw: clean up some compiler warnings

Reviewed-by: Ali Maredia <amaredia@redhat.com>

cmake: libneoradostest-support should be static

Some builds seem to be creating a libneoradostest-support.so dynamic
library instead, which results in unresolved dependencies when installing
ceph-test.

See: https://tracker.ceph.com/issues/38611
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cmake: enable write-back cache in spec

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

qa/suites/rados/singletone: whitelist MON_DOWN when injecting msgr errors

add a 'none.yaml' facet which doesn't whitelist so we hopefully capture
other causes of MON_DOWN.

This fixes at least one source of https://tracker.ceph.com/issues/45441

Fixes: https://tracker.ceph.com/issues/45441
Signed-off-by: Sage Weil <sage@newdream.net>

rgw: clean up some compiler warnings

Minimal changes to clean up a few compiler warnings.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #39561 into master

* refs/pull/39561/head:
qa/valgrind.supp: suppress free[] warnings
qa/suites: do not use notcmalloc flavor

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mon: fix INCOMPAT_QUINCY ondisk compatset feature bit

This was overlapping with pacific.

Fixes: https://tracker.ceph.com/issues/49143
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39501 from rhcs-dashboard/fix-tooltip-position

mgr/dashboard: fix tooltip for Provisioned/Total Provisioned fields

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge PR #38763 into master

* refs/pull/38763/head:
doc/cephfs: improve fs-nfs-exports.rst
doc/cephfs: improve nfs.rst

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

librbd/mirror: leave non-primary snapshot images in creating state

The creating state is a special case in rbd-mirror where it will
automatically delete the image since it assumes it's malformed.
A non-primary, snapshot-based mirror image needs to have at least
one non-primary snapshot and the first one is not created until
after replay has started. Now rbd-mirror will update the mirror
image state to the enabled state after creating the first
non-primary snapshot but before attempting the sync.

Fixes: https://tracker.ceph.com/issues/49238
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: ensure that the last non-primary snapshot cannot be pruned

Tweak the normal pruning behavior to ensure that an incomplete initial
non-primary snapshot is not included in the prune set since we know
it will be complete since otherwise the image would have been deleted
due to not updating the mirror-image-state to enabled. Also ensure
we cannot prune a non-primary mirror snapshot if we don't have a
predecessor.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39575 from zdover23/wip-doc-dev-t8y-intro-remove-upgrade-link-19-Feb-2021

doc/dev: intro: removing ceph-deploy mention

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39569 from yanghonggang/wip-multisite-etag-misidentify

rgw: multisite: fix single-part-MPU object etag misidentify problem

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/dev: intro: removing ceph-deploy mention

This removes a mention of the ceph-deploy manpage.
We also no longer direct people to use horses as
a primary form of transportation.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

librbd: memory is not preallocated in rbd c++ read api

Signed-off-by: weixinwei <weixw3@lenovo.com>

Merge PR #39566 into master

* refs/pull/39566/head:
cmake: build static libs if they are internal ones

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39472 from sebastian-philipp/doc-cephadm-host-labels

doc/orch: Host labels

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39571 from sebastian-philipp/doc-remove-upstart-from-freebsd

doc/install: Remove Upstart from Freebsd deployment

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #39546 from zdover23/wip-doc-dev-teuthology-intro-suites-inventory-rewrite-18-Feb-2021

doc/dev: t5logy: rewrite "suites inventory"

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/install: Remove Upstart from Freebsd deployment

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

rpm: openSUSE Tumbleweed has lua54 now

Until someone figures out an easier way, we are stuck with the minor Lua version
in the package names.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: set security headers

Fixes: https://tracker.ceph.com/issues/49243
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #39406 from weixinwei/master

os/bluestore/BlueFS: use iterator_impl::copy instead of bufferlist::c_str() to avoid bufferlist rebuild

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #39471 from sebastian-philipp/rados-rm-manage-upstart

doc/rados/operations: Remove upstart

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #39565 from badone/wip-fix-json-calls

test/osd: Use get_data() to simplify calls

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw multisite: fix single-part-MPU object etag misidentify problem

The single-part-MPU object is treated as a Atomic object.
Because its part_size is 0.

As MPU object's start_part_num is not 0 which can be used to
distinguish MPU object and Atomic object.

Fixes: https://tracker.ceph.com/issues/49357
Signed-off-by: Yang Honggang <yanghonggang@kuaishou.com>

librbd/cache/pwl: wait for the completion of reads

Shut down PWL after the read IOs are completed.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

cmake: build static libs if they are internal ones

there are chances that user or build script set `BUILD_SHARED_LIBS`,
so these convenience libraries (using the autotools' terminology)
are built and linked by never get installed.

Fixes: https://tracker.ceph.com/issues/38611
Fixes: https://tracker.ceph.com/issues/49080
Signed-off-by: Kefu Chai <kchai@redhat.com>

test/osd: Use get_data() to simplify calls

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #39556 from zdover23/wip-doc-dev-t8y-intro-how-integration-tests-are-run-rewrite-2021-Feb-19

doc/dev: t8y intro: edit "how to run int. tests"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39554 from zdover23/wip-doc-dev-t7gy-intro-testing-priority-n-2021-Feb-19

doc/dev: t7gy s/Priority/N/

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39293 from mcv21/rgw-admin-error-message

rgw/radosgw-admin clarify error when email address already in use

Merge pull request #38917 from joke-lee/rgw_sts_get_session_token_check_duration_always_failed

rgw: fix sts get_session_token duration check failed

qa/valgrind.supp: suppress free[] warnings

This is apparently not going to get fixed any time soon.

https://github.com/gperftools/gperftools/issues/792

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #39304 into master

* refs/pull/39304/head:
qa/tasks/ceph: set ctx.ceph[cluster_name].fsid
qa/tasks/rgw: pass fsid to rgw daemon
qa/suites/upgrade/octopus-x: reenable ragweed
qa/suites/upgarde/octopus-x: rename yaml file

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

doc/dev: t8y intro: edit "how to run int. tests"

This PR gives a bit more form to the section called
"how integration tests are run".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/dev: t7gy s/Priority/N/

"N" just looks better in this table than does
"Priority". I've been thinking about this all
day, and I just decided to make this change.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

qa/suites: do not use notcmalloc flavor

teuthology now knows how to run valgrind against a tcmalloc binary

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39548 from pkubaj/master

crc32c: fix build on ppc64le with clang

Reviewed-by: Kefu Chai <kchai@redhat.com>

crc32c: fix build on ppc64le with clang

Ceph uses assembly files from https://github.com/antonblanchard/crc32-vpmsum
Update them to the newest versions, which fixes build with Clang.

Signed-off-by: Piotr Kubaj <pkubaj@FreeBSD.org>

doc/orch: Host labels

Fixes: https://tracker.ceph.com/issues/45833
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38881 from ceph/default_images

mgr/cephadm: Put together default container images references

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39330 from sebastian-philipp/orch-rm-apply-funcs

mgr/orch: remove a bunch of duplicated code

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

doc/dev: t5logy: rewrite "suites inventory"

This really just re-organizes this information into
a table. I think tables are easier to read than what
was here before.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39542 from tchaikov/wip-devicehealth-flake8

mgr/devicehealth: silence flake8 warnings

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39514 from ceph/fix_buffer_overflow

rgw: fix buffer overflow in s3 select op

Merge pull request #39356 from xxhdx1985126/wip-crimson-client-request-idempotent

crimson/osd: make client_requests idempotent

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: make client_requests idempotent

When redoing client requests, they might have already taken effect
on the underlying disk. This commit deals with that situation by
making those requests immediately reply to clients if they are already
done

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

mgr/devicehealth: silence flake8 warnings

turns out 4840507cfcdd5182003671994d0bc9604d072e3e was racing with
99805d9cff87f7fe264bbadc235cb8cc311fd460. so we failed to identify this
before merging the former.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39413 from ronen-fr/wip-ronenf-blue-log

os/bluestore: reduce number of wasted checks for low-priority logs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #39365 from liewegas/mgr-ssd-used

mgr/devicehealth: extract and present SSD wear %

Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>