mgr/dashboard: Fix duplicate error messages

Duplicate error messages currently appear if the task wrapper service is
used. It calls 'notifyTask' on a failed task, this would be fine if
we didn't have the API interceptor, which watches all API requests and
triggers 'notifyTask' itself if an error appears.

Fixes issue #25139

Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #23281 from tspmelo/wip-opaque

mgr/dashboard: Fix localStorage problem in Jest

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix localStorage problem in Jest

Latest version of Jest was showing the following error:
"SecurityError: localStorage is not available for opaque origins"

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #23254 from tchaikov/wip-cmake-cleanup

cmake: modularize src/perfglue

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23255 from tchaikov/wip-seastar-yaml-cpp

seastar: lower the required yaml-cpp version to 0.5.1

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23253 from smithfarm/wip-doc-dev-toc

doc: make it easier to reach the old dev doc TOC

Reviewed-by: Dan Mick <dan.mick@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/dev/internals: drop :orphan: directive

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23165 from smithfarm/wip-doc-snaptrim-nit

doc/rados/operations/pg-states: fix PG state names, part 2

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #23072 from fullerdj/wip-djf-24687

mon/OSDMonitor: Warnings for expected_num_objects

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23268 from ceph/simple-testinfra-version

ceph-volume: do not pin the testinfra version for the simple tests

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: do not pin the testinfra version for the simple tests

For testinfra to work with a 2.6.x version of ansible it must use the
latest version.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

mon/OSDMonitor: Warn if missing expected_num_objects

When creating a pool on filestore, warn if the user appears to be
creating a pool to store a large number of objects but omitted the
expected_num_objects parameter. Create the pool anyway.

Fixes: http://tracker.ceph.com/issues/24687
Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #23263 from ceph/simple-ansible-version

ceph-volume: update version of ansible to 2.6.x for simple tests

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: update version of ansible to 2.6.x for simple tests

ceph-ansible now requires a 2.5.x or 2.6.x version of ansible if you're
using the master branch. This updates our functional tests for the
simple subcommand to use a 2.6.x version of ansible.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23256 from trociny/wip-25114

test/pybind: fix test_rbd.TestClone.test_trash_snapshot

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23168 from tchaikov/wip-dpdk-fixes

msg/async: fix FTBFS of dpdk

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23200 from votdev/bug_25068

mgr/dashboard: RGW is not working if an URL prefix is defined

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #22906 from tspmelo/wip-improve-summary

mgr/dashboard: Improve SummaryService and TaskWrapperService

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

do_cmake: fail if cmake fails

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add Findfmt.cmake

the cmake config provided by fmt-devel-3.0.2 does not offer fmt::fmt target,
so we need to offer a Findfmt.cmake to do this.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: make it easier to reach the old dev doc TOC

Signed-off-by: Nathan Cutler <ncutler@suse.com>

test/pybind: fix test_rbd.TestClone.test_trash_snapshot

Fixes: http://tracker.ceph.com/issues/25114
Signed-off-by: Mykola Golub <mgolub@suse.com>

cmake: make cpu_profiler a library target

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: make heap_profiler a library target

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: import allocator libraries in a better way

* instead expose the paths, it'd better to expose a library target.
* remove HAVE_LIBJEMALLOC, as it not used anywhere.

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: lower the required yaml-cpp version to 0.5.1

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23191 from dillaman/wip-23398

rbd: improved trash snapshot namespace handling

Reviewed-by: songweibin <song.weibin@zte.com.cn>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #23228 from tchaikov/wip-ceph-dencoder

test: move ceph-dencoder to src/tools

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20366 from Synesis-LLC/master

doc: print pg peering in SVG instead of PNG

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23248 from linuxbox2/rgw-fix-relaxed

rgw: fix build

Merge pull request #22758 from ricardoasmarques/wip-user-management

mgr/dashboard: Ceph dashboard user management from the UI

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

rgw: fix build

merge crossed with #23074

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #23247 into master

* refs/pull/23247/head:
Revert "rpm,deb: chown bluestore block* links to ceph.ceph on install/upgrade"
Revert "rpm,deb: fix chown to use -h"
rpm,deb: fix chown to use -h
rpm,deb: chown bluestore block* links to ceph.ceph on install/upgrade

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Revert "rpm,deb: chown bluestore block* links to ceph.ceph on install/upgrade"

This reverts commit 58cde3fd793c037beca6645cc30b2ae1c30e2af1.

This was only needed for luminous and mimic.  We are keeping these commits in master (and reverting them) only so that the cherry-pick -x references work.

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "rpm,deb: fix chown to use -h"

This reverts commit 92207c76b91c26c0558688a358ade7c0d94d1b59.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23234 from ceph/wip-rm24784

ceph-volume: PVolumes.get() should return one PV when using name or uuid

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23237 from neha-ojha/wip-whitelist-slow-requests

qa/tasks/thrashosds-health.yaml: whitelist slow requests

Merge pull request #23141 from alfredodeza/wip-rm24993

ceph-volume process.call with stdin in Python 3 fix

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #22533 from linuxbox2/wip-rgw-24507

rgw: add option for relaxed region enforcement

qa/tasks/thrashosds-health.yaml: whitelist slow requests

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #23074 from tchaikov/wip-seastar-config

common/config: add ConfigProxy for crimson

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/dashboard: Ceph dashboard user management

Fixes: https://tracker.ceph.com/issues/24446
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Skip tab focus on "show/hide password" button

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Add 'onCancel' output for 'ConfirmationModal'

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Add cd-select-badges component

Signed-off-by: Ricardo Marques <rimarques@suse.com>

ceph-volume: PVolumes.get() should return one PV when using name or uuid

It is possible to get duplicated pv entires from the 'pvs' lvm command.
If we're using PVolumes.get() with either pv_name or pv_uuid we can
safely return a single PVolume object. However, if we use pv_tags with
PVolumes.get() we must still raise a MultiplePVsError if many pvs
are found that have the tags, because they are not guaranteed to be the
same pv, as would be the case with pv_name or pv_uuid.

Fixes: http://tracker.ceph.com/issues/24784
Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #23232 into master

* refs/pull/23232/head:
mon: don't require CEPHX_V2 from mons until nautilus

ceph-volume tests run to log_descriptors, verifying utf-8

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume process decode to utf-8 when logging descriptors

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.process ensure that stdin works on py3 and py2.7

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume process encode stdin with utf-8 to prevent bytes TypeError

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mon: don't require CEPHX_V2 from mons until nautilus

The mimic 13.2.0 didn't have it.

Fixes: 3dc80e5f9b6ebf1bc1cecbd95b288005216bdbec
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23231 from tchaikov/wip-fio

tests/fio: fix build failures and ensure this is covered by run-make-check.sh

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

tools/ceph_dencoder: use full path for including headers

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Add match validator

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Add user management API

Fixes: https://tracker.ceph.com/issues/24446
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Add asserError helper method

Signed-off-by: Ricardo Marques <rimarques@suse.com>

cmake: sync up with upstream FIO

to pick up the change to FTBFS with C++ engine.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Improve SummaryService and TaskWrapperService

When you create a new task, and it stays running, it will be added automatically
to the summary data.

This will allows for us to deal with it more quickly, by subscribing
to the summaryService, and removes the need to pass a runningTasks array
between services.

Added 3 new methods to SummaryService.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22695 from a2batic/scss

mgr/dashboard: Color variables for color codes

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23100 from adamemerson/wip-robustly-notify

rgw: Robustly notify

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #23183 from ricardoasmarques/wip-fix-url-prefix

mgr/dashboard: Fix cherrypy static content URL prefix config

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Color variables for color codes
Assign variables to colors present in css files
Changed navigation icons padding because the notification dropdown was going out of window.
Fixes: http://tracker.ceph.com/issues/24575
Signed-off-by: Kanika Murarka <kmurarka@redhat.com>

Merge pull request #22857 from tspmelo/wip-move-test-file

mgr/dashboard: Move unit-test-helper into the new testing folder

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #22624 from ashutosh-narkar/opa-ceph

rgw: Initial work for OPA-Ceph integration

test: move ceph-dencoder to src/tools

as it belongs to ceph-common, not ceph-test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #23220 into master

* refs/pull/23220/head:
osd/OSDMap: CRUSH_TUNABLES5 added in jewel, not kraken

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/dashboard: RGW is not working if an URL prefix is defined

Fixes https://tracker.ceph.com/issues/25068

Signed-off-by: Volker Theile <vtheile@suse.com>

Initial work for OPA-Ceph integration

Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>

common/config_proxy: document Config{Values,Proxy}

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: update test_alien_echo.cc with f80b848d

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build c-ares if it's not found or not new enough

Signed-off-by: Kefu Chai <kchai@redhat.com>

c-ares: add c-ares submodule

c-ares is a C library for asynchronous DNS requests, seastar requires
c-ares 1.13.0 or up. but xenial and EPEL offer 1.10.0, so we need to
build and ship it as part of Ceph.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: add ceph::common::ConfigProxy

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: change md_config_obs_impl's template param to ConfigProxy

we were using LockPolicy as this tempalte class's template parameter,
but turns out ConfigProxy is a better parameter for it. as ConfigProxy is
"closer" to md_config_obs_impl than LockPolicy is.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: change md_config_impl<> back to plain class

now that all of the variables that change depending on seastar/alien
have been extracted out of md_config_impl<>, there is no need to make it
a template anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm,install-deps,cmake: install {c-ares,fmt}-devel

* ceph.spec.in: c-ares-devel and fmt-devel are installed as
  dependencies fo seastar.
* install-deps.sh: libfmt-dev is missing in xenial, so we only
  install libc-ares-dev for ubuntu in install-deps.sh .
* cmake: build libfmt if it's not found.

Signed-off-by: Kefu Chai <kchai@redhat.com>

fmt: add libfmt submodule

libfmt is a library for fast printf() support in C++, it is used by
seastar. but it's missing in xenial. so we need to ship it in our dist
tarball and build if if it's not found in system.

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: use the modernized cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: sync with upstream

to pick up the commit of 67bb775e, which marks
foreign_ptr::get_owner_shard() const.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: mark do_argv_commands() const

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: extract ObserverMgr from md_config_impl<>

crimson will have sharded<PGShard> (name may vary in future), and
underlying ObjectStore instances. they will be consumers of crimson
ConfigProxy, and will register them as an observer of config changes.
but as always, in crimson osd, world is sharded, each ConfigProxy will
only take care of the sharded services living on the shard which it is
in charge of. in other words, unlike alien's ConfigProxy, the
"observers" data structure is not shared, they are sharded as well. so
we need to extract the observers out of md_config_impl<>, and move it
into ConfigProxy.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/OSDMap: CRUSH_TUNABLES5 added in jewel, not kraken

In 0ceb5c0f3694725c7043ea99b077816da2ca44a7 I screwed up the version that
CRUSH_TUANBLES5 (chooseleaf_stable) was added: it comes from
043a7378ec4e828f2568ea3906d9076848161ccd, v10.0.3, which leads up to
jewel.

This, in turn, means that when jewel tunables are set we do not actually
encode the jewel tunables.  At least, luminous doesn't; jewel uses a
different code path to decode how to encode the OSDMap.

This, in turn, leads to crc errors:

- luminous mons running
- require_osd_release is jewel
- jewel tunables are set.  mon in-memory crushmap gets chooseleaf_stable=1
- osds are luminous, but don't see the new field in the crushmap
- upgrade finishes, and require_osd_Release becomes luminous
- the full osdmap encodes the chooseleaf_stable field, finally, and it
  is 1
- the osds try to encode the same thing, but their crushmap has 0.
-> crc mismatch

This is a regression from 0ceb5c0f3694725c7043ea99b077816da2ca44a7,
backported to luminous in 686b054f48c50859bd5e8bad93007bf9d7d7899c in
12.2.6.

Fixes: 686b054f48c50859bd5e8bad93007bf9d7d7899c
Fixes: http://tracker.ceph.com/issues/25057
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: prettify code

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Move unit-test-helper into the new testing folder

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22996 from batrick/mds-state-dot

doc: add mds state transition diagram

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22581 from fullerdj/wip-djf-24544

rgw: change default rgw_thread_pool_size to 512

mds: remove ascii art version of mds state diagram

Now refer to the user documentation. Any updates should go there!

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: create mds state documentation

Fixes: http://tracker.ceph.com/issues/22989
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20330 from joke-lee/wip-rgw-postobjloc

rgw: return valid Location element, PostObj

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #22937 from tianshan/fix_index_suggest_ut_fail

rgw: continue enoent index in dir_suggest

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #23095 from trociny/wip-24898

 librbd: ensure exclusive lock acquired when removing sync point snapshots

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23167 from tchaikov/wip-tools-rbd-cleanup

tools/rbd: assert(g_ceph_context) not g_conf

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22419 from votdev/hide_progress_bar

mgr/dashboard: Hide progress bar in case of an error

Merge pull request #23182 from alfredodeza/wip-ansible-26

ceph-volume tests/functional use Ansible 2.6

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23132 from dillaman/wip-25000

librbd: deep-copy should not write to objects that cannot exist

Reviewed-by: Mykola Golub <mgolub@suse.com>

qa/workunits/rbd: updates to the clone v2 CLI tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: clone detach should attempt to remove trashed snapshots

Fixes: http://tracker.ceph.com/issues/23398
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new helper method for retrieving a single snapshot info record

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: reduce log level of write denied permission error

The watch OSD op requires write permission since -EPERM will still
be returned to the user.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>