doc/rados/operations: enhance primary affinity section

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #38073 from badone/wip-use-right-tunable-for-pg-scrub-doc

doc: Use correct tunable for PG_NOT_SCRUBBED

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37883 from diwilli/patch-1

cephadm: Set listen-addresses on alertmanager container

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #38011 from mgfritch/cephadm-sudo-log-dir

cephadm: verify root privileges before log dir create

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge pull request #38057 from zdover23/wip-doc-dev-guide-essentials-source-clean-update

doc/dev: update essentials.rst

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38036 from zdover23/wip-doc-dev-guide-basic-workflow-make-check-nov-2020

doc/dev: improve make check text

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37977 from dillaman/wip-48145

librbd: update hidden global config when removing pool config override

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37975 from dang/wip-dang-webdir-empty-object

RGW - Don't create empty obj in webdir

Merge pull request #38060 from rhcs-dashboard/wip-master-auto_labels

github: fix autolabel paths

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

github: fix autolabel paths

Remove starting / from paths and add missing doc path from dashboard.

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #38055 from tchaikov/wip-doc-conf

doc/rados/configuration: document option types

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

doc/dev: update essentials.rst

This commit changes the noun "backup" to the
intended verb "back up" (this is one of my
technical documentation pet peeves and I couldn't
resist), and it also removes the verb "to nuke"
and replaces it with something less poetic but
more likely to be discoverable in a dictionary by
someone whose English is weak. It also adds a
missing colon.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/rados/configuration: document option types

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados: remove superfluous "::"

this change addresses a regression introduced by
0f2489406a8d4194c159c8777517d20b54e216b5

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: Use correct tunable for PG_NOT_SCRUBBED

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

doc/dev: improve make check text

This commit fixes the references to the
"make check" section in the "notes on
pr make check test" section and improves
the wording of that section so that it
reads a bit more clearly.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38049 from rhcs-dashboard/wip-master-auto_labels

github: autolabel PRs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38023 from zdover23/wip-doc-dev-guide-essentials-nov-2020

doc/dev: update essentials.rst nov 2020

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev: update essentials.rst

This commit improves the grammar and structure
of essentials.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #37902 from pdvian/wip-clog-health-detail

mon: Log "ceph health detail" periodically in cluster log

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

RGW - Don't create empty obj in webdir

Zipper objects shouldn't be empty, but should instead be null.  Fix this
case where parsing wasn't checking for empty object name, and the
website code that uses it.

Fixes: https://tracker.ceph.com/issues/48049
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

github: autolabel PRs

... starting with dashboard ones.

Requires https://github.com/marketplace/actions/labeler

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #38029 from kshtsk/wip-cephadm-bootstrap-remote

qa/task/cephadm: run cephadm only on bootstrap_remote

Merge pull request #36385 from ghyer/ghyer-dashboard

mgr/dashboard: Add 'Dirs' and 'Caps' for filesystems details

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38027 from s0nea/wip-dashboard-fix-doc-link

mgr/dashboard: fix HACKING.rst reference

Reviewed-by: aaSharma14 <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #37993 from votdev/issue_48051_ngx_pipe_function

mgr/dashboard: Use pipe instead of calling function within template

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37770 from pritha-srivastava/wip-sts-47871

rgw/sts: fix for validating path in role arn in assume role api.

mgr/dashboard: Use pipe instead of calling function within template

This PR introduces the 'ngx-pipe-function' package to call functions in templates.

Fixes: https://tracker.ceph.com/issues/48051
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #37918 from dungdm93/avoiding-relative-paths

mgr/dashboards: avoiding relative paths '../../..'

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: nizamial09 <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37922 from mgfritch/project-kubic

cephadm: install podman from the Kubic project

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38018 from tchaikov/wip-cmake-python

COPYING: note cmake/modules/FindPython* license and copyright

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38028 from tchaikov/wip-crimson-maybe-unused

crimson/os: mark unused variable [[maybe_unused]]

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #35865 from tchaikov/wip-crimson-recovery-qa

qa/suites: add crimson-rados suite

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #36759 from dzafman/wip-46264

mon: check for mismatched daemon versions

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mon: Log "ceph health detail" periodically in cluster log

change mon_health_to_clog_interval from 1_hr -> 10_min to
log health summary or detail more frequently.

Fixes: https://tracker.ceph.com/issues/48042
Signed-off-by: Prashant Dhange <pdhange@redhat.com>

test: Changes based on code review comments

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Apply suggestions from code review

Co-authored-by: Neha Ojha <nehaojha88@gmail.com>
Co-authored-by: Josh Durgin <jdurgin@redhat.com>
Signed-off-by: David Zafman <dzafman@redhat.com>

osd test: Delay reporting until mon_warn_older_version_delay has passed

Move release notes description to 16.0.0 and update
Update documentation

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Temporary fix for alienstore which doesn't support asok commands yet

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Document health check DAEMON_OLD_VERSION

Signed-off-by: David Zafman <dzafman@redhat.com>

Update Release notes

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Simple test case for version health warning

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Use a config variable debug_version_for_testing

Signed-off-by: David Zafman <dzafman@redhat.com>

mon: Add mds version collector

Signed-off-by: David Zafman <dzafman@redhat.com>

mon: Add mgr version collector

Signed-off-by: David Zafman <dzafman@redhat.com>

mon: Add osd version collector

Signed-off-by: David Zafman <dzafman@redhat.com>

mon: Verify version consistency across daemons

Add a function called check_daemon_versions()
Made minor changes based on pull request comments
Changed the way versions are checked along with some other minor modifications
Made minor modification to the error message
Modifications to include the hostname in the error message

Fixes: https://tracker.ceph.com/issues/46264
Signed-off-by: Tyler <tyler_sheehan@student.uml.edu>
Remove time since health errors aren't preserved if not set every time
Rename health error to DAEMON_OLD_VERSION
Reoganize code so that we can get versions from other daemon types
Change to map of version (key), a list of daemons (value)
Get actual daemon names from get_versions() requests
Fix message output based on how many daemons and how many versions
Add a config to be able to turn off this features
Give HEALTH_ERR if more than one older version exists
Fix minor indentation problem with health detail

Signed-off-by: David Zafman <dzafman@redhat.com>

qa/task/cephadm: run cephadm only on bootstrap_remote

Because not all the node in teuthology config might
have any cephadm related role we should not execute
cephadm on random node.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge PR #37978 into master

* refs/pull/37978/head:
qa: fix broken links in multimds suite

Reviewed-by: Jeff Layton <jlayton@redhat.com>

cephadm: verify root privileges before log dir create

root privileges need to be verified before creating the log dir

introduced by 81a7df0

Fixes: https://tracker.ceph.com/issues/48166
Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #37235 from adk3798/4mon

mgr/cephadm: only deploy 3 mons by default when there are 4 hosts

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

crimson/os: mark unused variable [[maybe_unused]]

silences the warning when building with -DNDEBUG:

segment_cleaner.h:399:10: warning: unused variable ‘ret’ [-Wunused-variable]
  399 |     auto ret = space_tracker->allocate(
      |          ^~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix HACKING.rst reference

Update the documentation reference as the content
of the HACKING.rst file has been moved to the Ceph
developer documentation.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37916 from orozery/librbd-crypto-wait-for-copyup

librbd: wait for copyup in unaligned crypto write

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38017 from YanfeiXu/yann-dev

cmake: add support for python3.9

Reviewed-by: Kefu Chai <kchai@redhat.com>

COPYING: note cmake/modules/FindPython* license and copyright

these files were copied from CMake project for better python3 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38021 from tchaikov/wip-mgr-liburl3

pybind/mgr: add urllib3==1.25.11 to mgr/requirements.txt

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

pybind/mgr: add urllib3==1.25.11 to mgr/requirements.txt

in hope to address the failure of

ERROR: Cannot install -r requirements.txt (line 3) and kubernetes because these package versions have conflicting dependencies.

The conflict is caused by:
    kubernetes 12.0.0 depends on urllib3>=1.24.2
    requests 2.24.0 depends on urllib3!=1.25.0, !=1.25.1, <1.26 and >=1.21.1

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add support for python3.9

add support for python3.9.

Signed-off-by: Yanfei Xu <yanfei.xu@windriver.com>

Merge pull request #37775 from dzafman/wip-upgrade-mimic-fix

test: Remove mimic-x* upgrade testing in master

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #38006 from xxhdx1985126/wip-crimson-recovery-bug-fix2

crimson/osd: make load_obc_for_recovery pop up all errors

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: add tslint rule to prevent import with more than 1 parent e.g '../../foobar'

Signed-off-by: Đặng Minh Dũng <dungdm93@live.com>

mgr/dashboard: npm run fix

Signed-off-by: Đặng Minh Dũng <dungdm93@live.com>

mgr/dashboard: reconfig tslint import match rule for local module

Signed-off-by: Đặng Minh Dũng <dungdm93@live.com>

mgr/dashboard: refactor imports

Signed-off-by: Đặng Minh Dũng <dungdm93@live.com>

mgr/dashboard: move "tsconfig.app.json" to <root> folder

Signed-off-by: Đặng Minh Dũng <dungdm93@live.com>

mgr/dasbhoard: add shortcut

Signed-off-by: Đặng Minh Dũng <dungdm93@live.com>

crimson/osd: make load_obc_for_recovery pop up all errors

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #37943 from xxhdx1985126/wip-crimson-recovery-bug-fix

crimson: recovery bug fix

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37997 from rhcs-dashboard/48155-fix-api-test-full-health

mgr/dashboard: fix API test: test_full_health

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: fix API test: test_full_health

Fixes: https://tracker.ceph.com/issues/48155
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge PR #37942 into master

* refs/pull/37942/head:
ceph-volume: fix lvm help test
ceph-volume: add a unit tests to lvm batch
ceph-volume: fix lvm batch auto with full SSDs

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

crimson/osd: make non-first push ops' version to be the version of the recovering object

An object may take more than one push op to recover, those later push ops also need to
set their version accordingly

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: make PglogBasedRecovery op take recovering objs triggered elsewhere into account

PGRecovery::start_recovery_ops() should wait for all inflight recovery ops, whether they are
started by BackgroundRecovery or not, otherwise there may be circumstances in which BackgroundRecovery
keep recursively invoking its do_recovery when start_recovery_ops returns recovery done while there are
still missing objects.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #37941 from dzafman/wip-48077

Allowing scrub configs begin_day/end_day to include 7 and begin_hour/end_hour to include 24 is confusing

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37539 from zdover23/wip-doc-ceph-df-2020-oct

doc/rados: ceph df output update

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test: unittest_osdscrub: Remove invalid hour test and add day of week testing

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Eliminate day of weeek 7 and hour 24

Add test case for permitted hours to make sure scrub doesn't start
Remove permitted hours in extended sleep test

Fixes: https://tracker.ceph.com/issues/48077
Signed-off-by: David Zafman <dzafman@redhat.com>

doc/rados: ceph df output update

This commit updates the "ceph df" output
so that it is current as of October 2020.
-Add correctly formatted `ceph df` output.
-Add explanation of "DIRTY" column.
-(DATA) remains to be defined (1 instance)
-(OMAP) remains to be defined (1 instance)
-USED remains to be defined (1 instance)
-Update prompts in "Checking OSD Status"

Fixes: https://tracker.ceph.com/issues/47523
Signed-off-by: Zac Dover <zac.dover@gmail.com>

cephadm: install podman from the Kubic project

ppa:projectatomic is no longer maintained, updates are now provided via
the Kubic project

Fixes: https://tracker.ceph.com/issues/48072
Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #37932 into master

* refs/pull/37932/head:
src/script/ceph-backport.sh: abort if jq not available

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume: fix lvm help test

ed5ceb0 changed the LVM help code but not the associated test.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

ceph-volume: add a unit tests to lvm batch

This commit adds unit tests in order to cover `_sort_rotational_disks()`
call when deploying with full hdd/ssd or mixed hdd/sdd scenarios.

Fixes: https://tracker.ceph.com/issues/48150
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
Co-authored-by: Dimitri Savineau <dsavinea@redhat.com>

ceph-volume: fix lvm batch auto with full SSDs

The ceph-volume lvm batch --auto introduced by [1] breaks the backward
compatibility when using non rotational devices only (SSD and/or NVMe).
Those devices are reaffected as bluestore db or filestore journal
devices while we want them as data devices.

Fixes: https://tracker.ceph.com/issues/48106
[1] https://github.com/ceph/ceph/pull/34740

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #37928 from tchaikov/wip-qa-ceph-ctx-archive

qa/tasks/ceph: extract update_archive_setting()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37453 from tspmelo/wip-rbd-change-detection

mgr/dashboard: Improve Change Detection on RBD Snapshot

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #37641 from henyxia/dev/henyxia/add-mgr-dashboard-prom-ssl-verify

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37983 from tchaikov/wip-crimson-os-co-locate-alien-allocator

crimson/os: do not configure seastar allocator for alien threads

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #32777 into master

* refs/pull/32777/head:
common,log,msg: Fix win32 compiler warnings
win32,common: provide a default ceph.conf location
common: add newline before parsing config on Windows
common: verify unix sockets support on Windows
common: fix md_config_t::get_cluster_name
global, common: Print config parse errors
common: accept config files containing whitespaces
librbd: avoid broken mingw rng
common: define DEV_NULL
tools,common,msg: Use O_BINARY for win32 compatibility
log: fix timestamps on Windows
common,librbd: add portable ceph_memzero_s
common,msg: Fix socket handling
common: Extend win32 compatbility layer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37984 from tchaikov/wip-fio

cmake: drop WITH_SYSTEM_FIO option

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #37965 from rhcs-dashboard/about-modal-cleanup

mgr/dashboard: Moving copyright variable into the app.constants

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37958 from ronen-fr/wip-ronenf-librbd-move1

librbd: removing a "pessimizing move"

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37990 from tchaikov/wip-liburing-fPIC

cmake: build liburing with -fPIC

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #37915 from gregsfortytwo/wip-stretch-fixes

Fix several stretch mode issues

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #37741 into master

* refs/pull/37741/head:
ceph-volume: remove mention of dmcache from docs and help text

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #37896 into master

* refs/pull/37896/head:
ceph-volume: implement the --log-level flag

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37506 from rhcs-dashboard/fix-47478-master

mgr/dashboard: Improvements in cluster > OSDs

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37649 from rhcs-dashboard/add-header-tooltip

mgr/dashboard: add description for CRUSH Ruleset

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

cmake: build liburing with -fPIC

in liburing,
https://github.com/axboe/liburing/commit/75cad68b953fe0fabbc4b1ced1ab4ba0ed5da18e
partially reverts
https://github.com/axboe/liburing/commit/4e360f71131918c36774f51688e5c65dea8d43f2,
which builds liburing.a with -fPIC.

so we need to pass -fPIC by ourselves. otherwise we'd have

/usr/bin/ld: ../../liburing/src/liburing.a(setup.ol): relocation R_X86_64_PC32 against symbol `io_uring_queue_mmap' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
src/test/fio/CMakeFiles/fio_ceph_objectstore.dir/build.make:154: recipe for target 'lib/libfio_ceph_objectstore.so' failed

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: expose FIO_INCLUDE_DIR via libfio

always use the fio built from source for better consistency.

Signed-off-by: Kefu Chai <kchai@redhat.com>