mgr/dashboard: update QoS values when editing a Pool/RBD image

Fixes: https://tracker.ceph.com/issues/47900
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #38029 from kshtsk/wip-cephadm-bootstrap-remote

qa/task/cephadm: run cephadm only on bootstrap_remote

Merge pull request #36385 from ghyer/ghyer-dashboard

mgr/dashboard: Add 'Dirs' and 'Caps' for filesystems details

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38027 from s0nea/wip-dashboard-fix-doc-link

mgr/dashboard: fix HACKING.rst reference

Reviewed-by: aaSharma14 <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #37993 from votdev/issue_48051_ngx_pipe_function

mgr/dashboard: Use pipe instead of calling function within template

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37770 from pritha-srivastava/wip-sts-47871

rgw/sts: fix for validating path in role arn in assume role api.

mgr/dashboard: Use pipe instead of calling function within template

This PR introduces the 'ngx-pipe-function' package to call functions in templates.

Fixes: https://tracker.ceph.com/issues/48051
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #37918 from dungdm93/avoiding-relative-paths

mgr/dashboards: avoiding relative paths '../../..'

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: nizamial09 <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37922 from mgfritch/project-kubic

cephadm: install podman from the Kubic project

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38018 from tchaikov/wip-cmake-python

COPYING: note cmake/modules/FindPython* license and copyright

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38028 from tchaikov/wip-crimson-maybe-unused

crimson/os: mark unused variable [[maybe_unused]]

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #35865 from tchaikov/wip-crimson-recovery-qa

qa/suites: add crimson-rados suite

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #36759 from dzafman/wip-46264

mon: check for mismatched daemon versions

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

test: Changes based on code review comments

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Apply suggestions from code review

Co-authored-by: Neha Ojha <nehaojha88@gmail.com>
Co-authored-by: Josh Durgin <jdurgin@redhat.com>
Signed-off-by: David Zafman <dzafman@redhat.com>

osd test: Delay reporting until mon_warn_older_version_delay has passed

Move release notes description to 16.0.0 and update
Update documentation

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Temporary fix for alienstore which doesn't support asok commands yet

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Document health check DAEMON_OLD_VERSION

Signed-off-by: David Zafman <dzafman@redhat.com>

Update Release notes

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Simple test case for version health warning

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Use a config variable debug_version_for_testing

Signed-off-by: David Zafman <dzafman@redhat.com>

mon: Add mds version collector

Signed-off-by: David Zafman <dzafman@redhat.com>

mon: Add mgr version collector

Signed-off-by: David Zafman <dzafman@redhat.com>

mon: Add osd version collector

Signed-off-by: David Zafman <dzafman@redhat.com>

mon: Verify version consistency across daemons

Add a function called check_daemon_versions()
Made minor changes based on pull request comments
Changed the way versions are checked along with some other minor modifications
Made minor modification to the error message
Modifications to include the hostname in the error message

Fixes: https://tracker.ceph.com/issues/46264
Signed-off-by: Tyler <tyler_sheehan@student.uml.edu>
Remove time since health errors aren't preserved if not set every time
Rename health error to DAEMON_OLD_VERSION
Reoganize code so that we can get versions from other daemon types
Change to map of version (key), a list of daemons (value)
Get actual daemon names from get_versions() requests
Fix message output based on how many daemons and how many versions
Add a config to be able to turn off this features
Give HEALTH_ERR if more than one older version exists
Fix minor indentation problem with health detail

Signed-off-by: David Zafman <dzafman@redhat.com>

qa/task/cephadm: run cephadm only on bootstrap_remote

Because not all the node in teuthology config might
have any cephadm related role we should not execute
cephadm on random node.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge PR #37978 into master

* refs/pull/37978/head:
qa: fix broken links in multimds suite

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #37235 from adk3798/4mon

mgr/cephadm: only deploy 3 mons by default when there are 4 hosts

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

crimson/os: mark unused variable [[maybe_unused]]

silences the warning when building with -DNDEBUG:

segment_cleaner.h:399:10: warning: unused variable ‘ret’ [-Wunused-variable]
  399 |     auto ret = space_tracker->allocate(
      |          ^~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix HACKING.rst reference

Update the documentation reference as the content
of the HACKING.rst file has been moved to the Ceph
developer documentation.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37916 from orozery/librbd-crypto-wait-for-copyup

librbd: wait for copyup in unaligned crypto write

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38017 from YanfeiXu/yann-dev

cmake: add support for python3.9

Reviewed-by: Kefu Chai <kchai@redhat.com>

COPYING: note cmake/modules/FindPython* license and copyright

these files were copied from CMake project for better python3 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38021 from tchaikov/wip-mgr-liburl3

pybind/mgr: add urllib3==1.25.11 to mgr/requirements.txt

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

pybind/mgr: add urllib3==1.25.11 to mgr/requirements.txt

in hope to address the failure of

ERROR: Cannot install -r requirements.txt (line 3) and kubernetes because these package versions have conflicting dependencies.

The conflict is caused by:
    kubernetes 12.0.0 depends on urllib3>=1.24.2
    requests 2.24.0 depends on urllib3!=1.25.0, !=1.25.1, <1.26 and >=1.21.1

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add support for python3.9

add support for python3.9.

Signed-off-by: Yanfei Xu <yanfei.xu@windriver.com>

Merge pull request #37775 from dzafman/wip-upgrade-mimic-fix

test: Remove mimic-x* upgrade testing in master

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #38006 from xxhdx1985126/wip-crimson-recovery-bug-fix2

crimson/osd: make load_obc_for_recovery pop up all errors

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: add tslint rule to prevent import with more than 1 parent e.g '../../foobar'

Signed-off-by: Đặng Minh Dũng <dungdm93@live.com>

mgr/dashboard: npm run fix

Signed-off-by: Đặng Minh Dũng <dungdm93@live.com>

mgr/dashboard: reconfig tslint import match rule for local module

Signed-off-by: Đặng Minh Dũng <dungdm93@live.com>

mgr/dashboard: refactor imports

Signed-off-by: Đặng Minh Dũng <dungdm93@live.com>

mgr/dashboard: move "tsconfig.app.json" to <root> folder

Signed-off-by: Đặng Minh Dũng <dungdm93@live.com>

mgr/dasbhoard: add shortcut

Signed-off-by: Đặng Minh Dũng <dungdm93@live.com>

crimson/osd: make load_obc_for_recovery pop up all errors

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #37943 from xxhdx1985126/wip-crimson-recovery-bug-fix

crimson: recovery bug fix

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37997 from rhcs-dashboard/48155-fix-api-test-full-health

mgr/dashboard: fix API test: test_full_health

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: fix API test: test_full_health

Fixes: https://tracker.ceph.com/issues/48155
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge PR #37942 into master

* refs/pull/37942/head:
ceph-volume: fix lvm help test
ceph-volume: add a unit tests to lvm batch
ceph-volume: fix lvm batch auto with full SSDs

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

crimson/osd: make non-first push ops' version to be the version of the recovering object

An object may take more than one push op to recover, those later push ops also need to
set their version accordingly

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: make PglogBasedRecovery op take recovering objs triggered elsewhere into account

PGRecovery::start_recovery_ops() should wait for all inflight recovery ops, whether they are
started by BackgroundRecovery or not, otherwise there may be circumstances in which BackgroundRecovery
keep recursively invoking its do_recovery when start_recovery_ops returns recovery done while there are
still missing objects.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #37941 from dzafman/wip-48077

Allowing scrub configs begin_day/end_day to include 7 and begin_hour/end_hour to include 24 is confusing

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37539 from zdover23/wip-doc-ceph-df-2020-oct

doc/rados: ceph df output update

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test: unittest_osdscrub: Remove invalid hour test and add day of week testing

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Eliminate day of weeek 7 and hour 24

Add test case for permitted hours to make sure scrub doesn't start
Remove permitted hours in extended sleep test

Fixes: https://tracker.ceph.com/issues/48077
Signed-off-by: David Zafman <dzafman@redhat.com>

doc/rados: ceph df output update

This commit updates the "ceph df" output
so that it is current as of October 2020.
-Add correctly formatted `ceph df` output.
-Add explanation of "DIRTY" column.
-(DATA) remains to be defined (1 instance)
-(OMAP) remains to be defined (1 instance)
-USED remains to be defined (1 instance)
-Update prompts in "Checking OSD Status"

Fixes: https://tracker.ceph.com/issues/47523
Signed-off-by: Zac Dover <zac.dover@gmail.com>

cephadm: install podman from the Kubic project

ppa:projectatomic is no longer maintained, updates are now provided via
the Kubic project

Fixes: https://tracker.ceph.com/issues/48072
Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #37932 into master

* refs/pull/37932/head:
src/script/ceph-backport.sh: abort if jq not available

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume: fix lvm help test

ed5ceb0 changed the LVM help code but not the associated test.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

ceph-volume: add a unit tests to lvm batch

This commit adds unit tests in order to cover `_sort_rotational_disks()`
call when deploying with full hdd/ssd or mixed hdd/sdd scenarios.

Fixes: https://tracker.ceph.com/issues/48150
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
Co-authored-by: Dimitri Savineau <dsavinea@redhat.com>

ceph-volume: fix lvm batch auto with full SSDs

The ceph-volume lvm batch --auto introduced by [1] breaks the backward
compatibility when using non rotational devices only (SSD and/or NVMe).
Those devices are reaffected as bluestore db or filestore journal
devices while we want them as data devices.

Fixes: https://tracker.ceph.com/issues/48106
[1] https://github.com/ceph/ceph/pull/34740

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #37928 from tchaikov/wip-qa-ceph-ctx-archive

qa/tasks/ceph: extract update_archive_setting()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37453 from tspmelo/wip-rbd-change-detection

mgr/dashboard: Improve Change Detection on RBD Snapshot

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #37641 from henyxia/dev/henyxia/add-mgr-dashboard-prom-ssl-verify

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37983 from tchaikov/wip-crimson-os-co-locate-alien-allocator

crimson/os: do not configure seastar allocator for alien threads

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #32777 into master

* refs/pull/32777/head:
common,log,msg: Fix win32 compiler warnings
win32,common: provide a default ceph.conf location
common: add newline before parsing config on Windows
common: verify unix sockets support on Windows
common: fix md_config_t::get_cluster_name
global, common: Print config parse errors
common: accept config files containing whitespaces
librbd: avoid broken mingw rng
common: define DEV_NULL
tools,common,msg: Use O_BINARY for win32 compatibility
log: fix timestamps on Windows
common,librbd: add portable ceph_memzero_s
common,msg: Fix socket handling
common: Extend win32 compatbility layer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37984 from tchaikov/wip-fio

cmake: drop WITH_SYSTEM_FIO option

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #37965 from rhcs-dashboard/about-modal-cleanup

mgr/dashboard: Moving copyright variable into the app.constants

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37958 from ronen-fr/wip-ronenf-librbd-move1

librbd: removing a "pessimizing move"

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37990 from tchaikov/wip-liburing-fPIC

cmake: build liburing with -fPIC

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #37915 from gregsfortytwo/wip-stretch-fixes

Fix several stretch mode issues

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #37741 into master

* refs/pull/37741/head:
ceph-volume: remove mention of dmcache from docs and help text

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #37896 into master

* refs/pull/37896/head:
ceph-volume: implement the --log-level flag

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37506 from rhcs-dashboard/fix-47478-master

mgr/dashboard: Improvements in cluster > OSDs

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37649 from rhcs-dashboard/add-header-tooltip

mgr/dashboard: add description for CRUSH Ruleset

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

cmake: build liburing with -fPIC

in liburing,
https://github.com/axboe/liburing/commit/75cad68b953fe0fabbc4b1ced1ab4ba0ed5da18e
partially reverts
https://github.com/axboe/liburing/commit/4e360f71131918c36774f51688e5c65dea8d43f2,
which builds liburing.a with -fPIC.

so we need to pass -fPIC by ourselves. otherwise we'd have

/usr/bin/ld: ../../liburing/src/liburing.a(setup.ol): relocation R_X86_64_PC32 against symbol `io_uring_queue_mmap' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
src/test/fio/CMakeFiles/fio_ceph_objectstore.dir/build.make:154: recipe for target 'lib/libfio_ceph_objectstore.so' failed

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: expose FIO_INCLUDE_DIR via libfio

always use the fio built from source for better consistency.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: drop WITH_SYSTEM_FIO option

there is no well-known distro packaging fio.h, so drop this option.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37981 from tchaikov/wip-make-dist

make-dist: add liburing to dist tarball

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

cmake: use make explicitly to build fio

we cannot assume that user uses "make" as the generator of cmake, if,
for instance, ninja is used, `$(MAKE)` is not a valid variable in the
generated `build.ninja`. so we should use "make" explicitly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not configure seastar allocator for alien threads

4cd2b00d2a703510777bd761609be221859bd790 allows us to colocate seastar
allocator used by seastar reactors and libc allocator used by alien threads,
there is no need to configure seastar allocator for alien thread
anymore.

this change partially reverts

- f7715b86ef09a820adb2d43d4e82cabc9a04199f
- b55919071fc872d880bebe70bc270b9d66d02ab0

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not #include unused headers

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use src/liburing for building liburing if it exists

so we can build with liburing enabled if the dist tarball contains
liburing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

make-dist: use a loop to collect the ingredients of the dist tarball

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

make-dist: add liburing to dist tarball

since liburing is enabled by default, let's included it in the dist
tarball.

Signed-off-by: Kefu Chai <kchai@redhat.com>

make-dist: extract download_from() out

so we can reuse it

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: fix broken links in multimds suite

Fixes: #37899
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36998 into master

* refs/pull/36998/head:
doc/mgr/volumes: Document authorize/deauthorize cli commands
qa/tasks/cephfs: test `fs subvolume authorize/deauthorize`
mgr/volumes: Allow/deny auth IDs access to FS subvolumes
mon/MonCap: allow 'profile mgr' to create/update/del auth IDs

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37618 into master

* refs/pull/37618/head:
mds: throttle cap acquisition via readdir

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37662 into master

* refs/pull/37662/head:
mon/MDSMonitor: deal with subscription after being expired

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37600 from mgfritch/cephadm-nfs-rgw

mgr/cephadm: add RGW support for NFS ganesha

Reviewed-by: Daniel-Pivonka <dpivonka@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37792 from p-se/monitoring-prom-many-to-many

mgr/cephadm: many-to-many Prometheus error

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #37967 from linuxbox2/wip-rgwfile-zwrite

rgw_file: fix some zipper flow for RGWLibContinuedReq

Merge pull request #37968 from changchengx/paxos

mon/PaxosService: remove unnecessary code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37970 from toabctl/fix-orch-cli-doc-link

mgr/orchestratori: Fix doc link in README.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix linting issues

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Add 'Dirs' and 'Caps' for filesystems details

Signed-off-by: Haoyu Gao <i@ghyer.com>

mgr/orchestrator_cli: Fix doc link in README.rst

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

doc: update stretch mode doc to not be a lie

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: include stretch mode status in more outputs

It was missing from the monmap dump, and included in OSDMap::dump()
but not the OSDMap::print() which is used for text-based CLI queries.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>