librbd/cache/pwl: wait for the completion of reads

Shut down PWL after the read IOs are completed.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #39556 from zdover23/wip-doc-dev-t8y-intro-how-integration-tests-are-run-rewrite-2021-Feb-19

doc/dev: t8y intro: edit "how to run int. tests"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39554 from zdover23/wip-doc-dev-t7gy-intro-testing-priority-n-2021-Feb-19

doc/dev: t7gy s/Priority/N/

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39293 from mcv21/rgw-admin-error-message

rgw/radosgw-admin clarify error when email address already in use

Merge pull request #38917 from joke-lee/rgw_sts_get_session_token_check_duration_always_failed

rgw: fix sts get_session_token duration check failed

Merge PR #39304 into master

* refs/pull/39304/head:
qa/tasks/ceph: set ctx.ceph[cluster_name].fsid
qa/tasks/rgw: pass fsid to rgw daemon
qa/suites/upgrade/octopus-x: reenable ragweed
qa/suites/upgarde/octopus-x: rename yaml file

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

doc/dev: t8y intro: edit "how to run int. tests"

This PR gives a bit more form to the section called
"how integration tests are run".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/dev: t7gy s/Priority/N/

"N" just looks better in this table than does
"Priority". I've been thinking about this all
day, and I just decided to make this change.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39548 from pkubaj/master

crc32c: fix build on ppc64le with clang

Reviewed-by: Kefu Chai <kchai@redhat.com>

crc32c: fix build on ppc64le with clang

Ceph uses assembly files from https://github.com/antonblanchard/crc32-vpmsum
Update them to the newest versions, which fixes build with Clang.

Signed-off-by: Piotr Kubaj <pkubaj@FreeBSD.org>

Merge pull request #38881 from ceph/default_images

mgr/cephadm: Put together default container images references

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39330 from sebastian-philipp/orch-rm-apply-funcs

mgr/orch: remove a bunch of duplicated code

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #39542 from tchaikov/wip-devicehealth-flake8

mgr/devicehealth: silence flake8 warnings

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39514 from ceph/fix_buffer_overflow

rgw: fix buffer overflow in s3 select op

Merge pull request #39356 from xxhdx1985126/wip-crimson-client-request-idempotent

crimson/osd: make client_requests idempotent

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: make client_requests idempotent

When redoing client requests, they might have already taken effect
on the underlying disk. This commit deals with that situation by
making those requests immediately reply to clients if they are already
done

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

mgr/devicehealth: silence flake8 warnings

turns out 4840507cfcdd5182003671994d0bc9604d072e3e was racing with
99805d9cff87f7fe264bbadc235cb8cc311fd460. so we failed to identify this
before merging the former.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39413 from ronen-fr/wip-ronenf-blue-log

os/bluestore: reduce number of wasted checks for low-priority logs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #39365 from liewegas/mgr-ssd-used

mgr/devicehealth: extract and present SSD wear %

Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

Merge pull request #39346 from tchaikov/wip-ipv6-lo

common/ipaddr: Allow binding ipv6 addr on lo

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39374 from tchaikov/wip-ceph-health-newline

mon/Monitor: add newline at end of output "ceph health -f json"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39391 from tchaikov/wip-mgr-more-flake8

pybind/mgr: add more modules to flake8

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39528 from zdover23/wip-doc-dev-teuthology-intro-testing-priority-2021-Feb-17

doc/dev: intro: rewrite "testing priority"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39516 from rzarzynski/wip-crimson-6threads-in-alienstore

crimson/osd: tune the default for crimson_alien_op_num_threads.

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39506 from tchaikov/wip-rpm-multi-thread-xz

ceph.spec.in: use multi-threaded xz compression and cleanups

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #39456 from cbodley/wip-qa-rgw-multisite-upgrade

qa/rgw: run multisite tests with mixed cluster versions

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

qa/tasks/ceph: set ctx.ceph[cluster_name].fsid

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rgw: pass fsid to rgw daemon

Fixes: https://tracker.ceph.com/issues/48754
Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/upgrade/octopus-x: reenable ragweed

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/upgarde/octopus-x: rename yaml file

We're installing octopus, not pacific

Signed-off-by: Sage Weil <sage@newdream.net>

doc/dev: intro: rewrite "testing priority"

This PR improves the English in the "Testing
Priority" section and makes a few structural
changes to make the intent clearer.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39363 from rhcs-dashboard/fix-orch-host-e2e

mgr/dashboard: check .badge instead of text for expected label

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge PR #38610 into master

* refs/pull/38610/head:
ceph-volume: pass --filter-for-batch from drive-group subcommand

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

qa/rgw: add pacific-x upgrade suite for multisite

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: add octopus-x upgrade suite for multisite

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: add 'branch' option to rgw-multisite-tests task

Fixes: https://tracker.ceph.com/issues/49330
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39467 from sebastian-philipp/doc-cephadm-mon-quorum

doc/cephadm: Restoring the MON quorum

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #39496 from liewegas/cephadm-refresh-health-update

mgr/cephadm: fix host refresh

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39372 from rhcs-dashboard/avoid-document-write

mgr/dashboard: avoid using document.write()

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39462 from rhcs-dashboard/fix-alerts-mtuMismatch

mgr/dashboard: fix MTU Mismatch alert

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

ceph.spec.in: use %make_install macro

hide buildroot in the implementation of this macro

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: use %make_build instead of "make"

so we can hide the `_smp_mflags` in `%make_build` macro

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: use a macro helper to limit number of jobs

so it can work with the rpm macros better.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: tune the default for crimson_alien_op_num_threads.

This commit tries to optimize the default size of AlienStore's
thread pool. Recent testing has shown the intial value, which
is `16`, is far from being optimal while the optimum is close
to `4`:

  https://gist.github.com/rzarzynski/c851212f4b0baef4097a6087533ba17b

Further testing by Mark Nelson suggests that the sweet spot is
closer to `8` taking into account 4 KB writes with lower queue
depths:

  https://docs.google.com/spreadsheets/d/1IR9ysWRkaGdX5e9w_YV8kfeeglR2pEY3V9hovY0ECV8/edit#gid=1884533182

As a kind of compromise `6` is proposed in this commit.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: check .badge instead of text for expected label

this change fixes a regression introduced by
8c5e31ec1a13bc53394eb2cb6880d74db169fac4 which broke the 01-hosts.e2e-spec.ts test
driven by test_dashboard_e2e.sh

Fixes: https://tracker.ceph.com/issues/49205
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #39464 from sebastian-philipp/doc-cephadm-unmanaged

doc/cephadm: Disable automatic deployment of daemons

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

there could a case where capacity is equal to string size, and cause invalid write(4 bytes); thus also the last 4-bytes are appended

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

Merge pull request #39459 from tchaikov/wip-doc-api-command

doc: wrap example command

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

ceph.spec.in: specify cmake variable type

for better readablity

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: use %cmake macro

%cmake passes a bunch of default flags, so we can save some efforts.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: use RPM_BUILD_NCPUS for number of jobs

this env variable is defined by /usr/lib/rpm/macros. let's used it
instead of inventing our own version.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: use multi-threaded xz compression

so we can build the compressed src rpm and binary rpms with smaller
latency.

see also
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.3_release_notes/technology_previews_installation_and_booting

by default, rpm uses

w9.gzdio

which means: gzip level 9. but since xz is able to compress using
multiple threads, let's switch to xz.

xz's default compression level is 6, but ceph's source and binary
packages are relatively large. let's use a higher compression level.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #39490 into master

* refs/pull/39490/head:
cephadm: fix selinux mount mis-indent

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge pull request #39497 from zdover23/wip-doc-dev-teuthology-intro-workbench-removal-16-Feb-2021

doc/dev: removing workbench docs (part 2)

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39499 from zdover23/wip-doc-dev-teuthology-teuthology-describe-tests-repair-2021-Feb-16

doc/dev: correct "teuthology-describe-tests"

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38030 from p-se/prom-alert-package-drops-leeway

mgr/dashboard: prometheus alerting: add some leeway for package drops and errors

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #39422 from kamoltat/wip-ksirivad-autoscaler-release-notes

PendingReleaseNotes: mgr/pg_autoscaler

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/dev: correct "teuthology-describe-tests"

This tiny commit does this:

s/teuthology-describe-tests/teuthology-describe/

The name of this feature was changed from "teuthology-
describe-tests" to "teuthology-describe", and this
commit brings up to date the title of the section
detailing it.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/dev: removing workbench docs (part 2)

This removes a paragraph in the section "How
Integration Tests Are Run" that described
ceph-workbench. ceph-workbench is deprecated,
so that paragraph is now gone.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

mgr/cephadm: fix host refresh

Fixes: 01f60cf4e0a751c314120c02956d4ff941eb71b4
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39323 from sebastian-philipp/cephadm-simplify-cephadmservice-config

mgr/cephadm: Add config() to CephadmService

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #39174 from zdover23/wip-doc-37949-grammar-edit

doc/dev: Teuthology guide PR#37949 grammar edit

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39392 from tschoonj/amqp-ssl

rgw: add support for SSL encrypted AMQP connections

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

doc/cephadm: Disable automatic deployment of daemons

Fixes: https://tracker.ceph.com/issues/45767
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/dev: Teuthology guide PR#37949 grammar edit

This PR improves the wording of the technical
information added to the documentation in PR#37949.
This is the second is a series of two PRs, which series
is dedicated to testing a workflow wherein developers
add technical information to the documentation and then
technical writers improve its presentation.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39430 from liewegas/bug-48142

qa/suites/rados/cephadm/upgrade: change starting version by distro

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39493 from rzarzynski/wip-crimson-cond-mon-config

crimson/osd: fetch configuration from monitors conditionally.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39492 from zdover23/wip-doc-dev-teuthology-workflow-cephfs-shell-typo-15-Feb-2021

doc/dev: cephfs-shell typo repair

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: prometheus alerting: add some leeway for package drops and errors (1%)

Fixes: https://tracker.ceph.com/issues/48201
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

crimson/osd: fetch configuration from monitors conditionally.

Before this change we were always attempting to fetch the config
from monitors, even if `--no-mon-config` had been passed. This
was the reason for failures at teuthology like:
http://pulpito.front.sepia.ceph.com/rzarzynski-2021-02-15_21:09:02-rados-master-distro-basic-smithi/5885250

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/cephadm: Restoring the MON quorum

Fixes: https://tracker.ceph.com/issues/49214
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39473 from sebastian-philipp/doc-cephamd-trouble-events

doc/cephadm: Per-service and per-daemon events

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

mgr/cephadm: _add_daemon: remove config_func

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add config() to CephadmService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: remove a bunch of duplicated code

`ceph daemon add [..]` are all the same thing

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: remove a bunch of duplicated code

We only need to call self.apply()

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39465 from sebastian-philipp/install-deps-rm-libboost1.72.list

install-deps: Cope with nonexistent /etc/apt/sources.list.d/ceph-libboost1.72.list

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39249 from varshar16/wip-fix-vstart-ganesha-error

vstart: fix ganesha cluster id and rados url errors

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

doc/dev: cephfs-shell typo repair

This corrects the formatting for a mention of
cephfs-shell. Previously, it was not properly
formatted. This commit adds two backticks before
the string "cephfs-shell", and adds to backticks
after it.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

cephadm: fix selinux mount mis-indent

We want to map this path even if we had to create the empty directory.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #39150 into master

* refs/pull/39150/head:
qa/suites/rados/cephadm/thrash: add cephadm thrashing test

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #39482 into master

* refs/pull/39482/head:
Revert "Merge pull request #37764 from mgfritch/cephadm-no-container-init"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #39424 into master

* refs/pull/39424/head:
cephadm: Mounting <empty> folder for selinux only if it is needed

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge PR #39185 into master

* refs/pull/39185/head:
os/bluestore: Add option to check BlueFS reads

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #39281 into master

* refs/pull/39281/head:
mgr/MgrClient; less copying of mgr report struct

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #39282 into master

* refs/pull/39282/head:
osdc/Objecter: remove pool_full_try interface
librados: set FULL_TRY flag on ioctx, not objecter
librados/IoCtxImpl: pass extra_op_flags everywhere

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #39283 into master

* refs/pull/39283/head:
qa/suites/rados/dashboard: whitelist TELEMETRY_CHANGED

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

Merge PR #39381 into master

* refs/pull/39381/head:
mgr/cephadm: based sleep interval on configured intervals/timeouts

Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: Mounting <empty> folder for selinux only if it is needed

There are OSs without </usr/share/empty> folder. And selinux can be
enabled or not.

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Revert "Merge pull request #37764 from mgfritch/cephadm-no-container-init"

This reverts commit f635555fe76322336a1e0e8cc3c199562642ad0e, reversing
changes made to d4d3d17b23fcf6d8808a96b992f56924f495b174.

This PR seems to be (indirectly?) responsible for
  https://tracker.ceph.com/issues/49237

Also, it was causing the rados.py task's follow-up step to wait
for snap trimming to fail: it would time out a 'ceph osd dump --format=json'
command.  :/

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39461 from yuvalif/wip-yuval-fix-49261

rgw/test: allow for duplicate events in pubsub

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/cephadm: Per-service and per-daemon events

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39386 from zdover23/wip-doc-dev-triggering-tests-rewrite-10-Feb-2021

doc/dev: teuthology: improve triggering tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev: teuthology: improve triggering tests

- This PR cleans up the language in the "Triggering
  Tests" section of the "Workflow" chapter of the
  "Tests: Integration Tests" (Teuthology) guide.
- Added bash prompts.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39407 from zdover23/wip-doc-dev-testing-qa-changes-rewrite-11-Feb-2021

doc/dev: rewrite 'testing qa changes'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev: rewrite 'testing qa changes'

This PR rewrites the "Testing QA Changes (Without
Re-Building Binaries)" section of the Developer
Guide. I also tried to unbury the lede.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge PR #38562 into master

* refs/pull/38562/head:
Update batch.py

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

mgr/dashboard: avoid using document.write()

Fixes: https://tracker.ceph.com/issues/49220
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

install-deps: Cope with nonexistent /etc/apt/sources.list.d/ceph-libboost1.72.list

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

rgw: add support for SSL encrypted AMQP connections

Fixes: https://tracker.ceph.com/issues/42902
Signed-off-by: Tom Schoonjans <Tom.Schoonjans@rfi.ac.uk>