rbd: delay parsing of default kernel map options

It now connects to the cluster to retrieve any global config
overrides before processing the default map options.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: 'list' action should connect to cluster before querying config

This will allow global config overrides for the max concurrent ops to
be applied prior to listing the images.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: drop querying of RBD default features

These will be computed internally by librbd using any potential
global configuration overrides.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: delay determination of default pool name

Wait until after the CLI has connected to the cluster and therefore
has received all global configuration overrides.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: let librados finish initializing the global context

librados will retrieve a bootstrap config update from the MONs
after connecting and then finish initializing the context. In
general, non-primitive config options cannot be updated after
initialization is complete. This prevents the rbd CLI from
supporting non-primitive config overrides from the MON.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon/MonClient: apply MON config before finishing bootstrap

In general, non-primitive config options cannot be updated after
the service threads have been marked as running. In librados, this
happens after the MonClient bootstrap process. However, the configs
received from the MON were applied in a separate thread which can
race with librados marking the service threads as running.

Now the received MON configs will be applied within the context of
the bootstrap function call instead of deferring to a separate
thread.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36476 from ceph/42072-landing-page

mgr/dashboard: Landing Page improvements

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36669 from smithfarm/wip-nautilus-11

doc/releases: add Nautilus 14.2.11 to timeline

Reviewed-By: Kefu Chai <kchai@redhat.com>
Reviewed-By: Zac Dover <zac.dover@gmail.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #36508 from mgfritch/cephadm-nfs-rados-pool

mgr/cephadm: check existence of the RADOS pool

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #36228 from vosdev/doc-mgr-crash-typo

doc/mgr/crash: Add missing command in rm example

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36623 from cheese/master

common: document the current semantics of bl::append(std::istream)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36418 from changchengx/osd_map

osd/OSDMap: corret osd_info_t last_clean comment

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36642 from tchaikov/wip-crimson-silence-warnings

crimson: silence warnings from clang++

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: do not capture unused variables

this change silences warnings like

src/crimson/osd/pg_backend.cc:232:34: warning: lambda capture 'this' is not used [-Wunused-lambda-capture]
        [&os, &osd_op, &extents, this](auto&& bl) -> read_errorator::future<> {

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not return a value for non-reachable branch

for silencing a warning like

btree/btree_lba_manager.cc:177:12: warning: binding dereferenced null pointer to reference has undefined behavior [-Wnull-dereference]
    return *static_cast<btree_range_pin_t*>(nullptr);
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

ceph_abort_msg() is marked [[noreturn]], so we can just use it here
without worrying about complains from compiler.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases: add Nautilus 14.2.11 to timeline

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #36479 from dillaman/wip-46669

test/rbd-mirror: pool watcher registration error might result in race

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #36639 from tchaikov/wip-crimson-build-push-op

crimson/osd: clear data_included if fiemap fails

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #36505 from rzarzynski/wip-crimson-bump-up-seastar

crimson: bump up seastar version to freshest master

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36529 from tchaikov/wip-crimson-dts-9

ceph.spec.in: use devtoolset-9 for building crimson

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

install-deps.sh: add tchaikov's copr repo for crimson build on el8

we need it before the updated version of gcc-toolset9 addressing
https://bugzilla.redhat.com/show_bug.cgi?id=1853900 is available in
RHEL8/CentOS8.

see https://copr.fedorainfracloud.org/coprs/tchaikov/gcc-toolset-9

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: use gcc-toolset-9 for building crimson

since seastar dropped support from C++14, we have to move to a compiler
with a decent C++17 support.

in this change, gcc-toolset-9 is used for compiling ceph if seastar is
enabled. use version >= 9.2.1-2.2, because of
https://bugzilla.redhat.com/show_bug.cgi?id=1853900

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: clear data_included if read fails

* extract read_object_for_push_op() out of build_push_op().
  build_push_op() is composed of multiple steps. would be better
  if we can split and extract them into separated method for better
  readability
* clear push_op->data_included() if fiemap fails. before this change,
  a local copy of `copy_subset` is cleared if fiemap fails, but
  push_op->data_included is not updated in this case. we should
  clear the latter if no data is returned in push_op->data. so,
  in this change, push_op->data_included is cleared if fiemap
  fails. the same applies to other cases where no data is returned in
  push_op
* add comments to outline the steps in read_object_for_push_op().

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: print detailed info of object being recovered

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: remove stale comment

CyanStore supports readv() now, so remove this comment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not use __func__ in a lambda

otherwise we would have logging messages like

operator(): returning obc ...

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36618 from tspmelo/wip-backport-jq

script/ceph-backport.sh: Check if jq is installed

Merge pull request #36609 from tspmelo/wip-doc-browsers

mgr/dashboard: Update list of supported browsers

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #36624 from lixiaoy1/fix_config

rbd/rwl: fix the config issue

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36466 from lixiaoy1/rbd_image_info_br

rbd/status: get image cache state

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36422 from orozery/object_dispatch_assert_version_write

librbd: support writing with version assertion on object dispatch

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36366 from runsisi/wip-fix-free

librbd: fix possible use after free error while shuting down exclusive lock

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36561 from lixiaoy1/fix_queue

rbd/rwl: fix the construction of ContextWQ

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Update list of supported browsers

Fixes: https://tracker.ceph.com/issues/46923
Signed-off-by: Tiago Melo <tmelo@suse.com>

rbd/rwl: fix the config issue

Include acconfig.h to the librbd/cache/Util.h.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

script/ceph-backport.sh: Check if jq is installed

If you tried to setup ceph-backport.sh without having 'jq' installed, you would
get a totally unrelated error message and it was hard to figure out the problem.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36604 from tspmelo/wip-favicon-circle

mgr/dashboard: Fix favicon white circle

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #36602 from tspmelo/wip-spell-check

mgr/dashboard: Fix spell errors

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #36585 from votdev/proper_format_portals

mgr/dashboard: Proper format iSCSI target portals

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36566 from votdev/issue_46900_autocomplete

mgr/dashboard: Strange iSCSI discovery auth behavior

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36567 from votdev/issue_46901_hide_limit_action_field

mgr/dashboard: Hide table action input field if limit=0

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

common: document the current semantics of bl::append(std::istream)

bl::append(std::istream) comes with a surprise implementation. People may think
it would just slurp the indentical binary data from the stream. Document the
trap to let people don't mis-use it.

Signed-off-by: Robin Lee <cheeselee@fedoraproject.org>

Merge pull request #36510 from athanatos/sjust/wip-tm-decref

crimson: remove extent from cache in TransactionManager::dec_ref

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore/transaction_manager: complete dec_ref

Previously, dec_ref didn't handle actually retiring the
extent from the cache.  dec_ref will now reach into the
cache and mark the extent retired if it exists either
in the cache or in the current transaction.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: clarify ref count operation return

Previously, we returned a refcount from inc_ref and dec_ref.  Now,
return the paddr as well for future code accounting for released
extents.

In addition, replumb btree_lba_manager to return an enoent error if
the mapping does not exist, and the resulting refcount, paddr
otherwise with a refcount of 0 indicating that the mapping has
been removed.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: don't mark invalid extents clean

In the event that an extent is created and removed in the same
transaction (invalid extent in fresh_block_list), update
block specific metadata but don't add to cache and especially
do not mark clean.  LBAManager::complete_commit implementations
are meant to use skip based on state in the same way.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cached_extent: add print_detail for logical extents

This way we can always see laddr in debug output.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #36591 from idryomov/wip-snap-create-progress

qa: update krbd tests for snap create

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36434 from ricardoasmarques/fix-iscsi-logged-in-client-edit

mgr/dashboard: Unable to edit iSCSI logged-in client

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #36564 from ceph/46899-log-info-from-unhandled-errors

mgr/dashboard: log useful information from internal server errors

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Fabrizio D'Angelo <fdangelo@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard: Landing Page improvements

Fixes: https://tracker.ceph.com/issues/42072
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

mgr/dashboard: Fix favicon white circle

A white circle was displayed when the user logged out.

Fixes: https://tracker.ceph.com/issues/46919
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix spell errors

Fixes: https://tracker.ceph.com/issues/46917
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Proper format iSCSI target portals

Proper format the 'Portals' and 'Images' columns.

Fixes: https://tracker.ceph.com/issues/46908
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #36532 from simon-rock/crash_on_restart

mds : move start_files_to_recover() to recovery_done

mds : move start_files_to_recover()  to recovery_done

the requests in the queue, named waiting_for_replay, may modify the state of the filelock,
resulting in the wrong lock state when repairing file (start_files_to_recover)

Fixes : https://tracker.ceph.com/issues/46906
Signed-off-by: Simon Gao <simon29rock@gmail.com>

Merge pull request #36119 from votdev/issue_44831_deploy_service

mgr/dashboard: Create Ceph services via Orchestrator by using ServiceSpec

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

qa: update krbd tests for snap create

"rbd snap create" now reports progress.  Pass --no-progress, as in
commit b5a5fea9e2b4 ("test/cli-integration/rbd: tweak after snap create
started to show progress").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #36569 from yuriw/wip-yuriw-crontab-master

qa/tests: use "-k distro" for all suites (except krbd)

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36570 from theanalyst/doc/releases/v14.2.11

doc: releases: notes for v14.2.11 nautilus

Reviewed-By: Neha Ojha <nojha@redhat.com>
Reviewed-By: Josh Durgin <jdurgin@redhat.com>
Reviewed-By: Patrick Donelly <pdonell@redhat.com>

rbd/rwl: fix the construction of ContextWQ

Fix the build issue caused by the commit c11594a which changed the constructor
of ContextWQ.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

mgr/dashboard: log useful information from internal server errors

Fixes: https://tracker.ceph.com/issues/46899
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

rbd/status: get image cache state

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

mgr/dashboard: Strange iSCSI discovery auth behavior

Disable 'autocomplete' for the user and password form fields.

Fixes: https://tracker.ceph.com/issues/46900
Signed-off-by: Volker Theile <vtheile@suse.com>

doc: releases: notes for v14.2.11 nautilus

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #36550 from tspmelo/wip-npm-update

mgr/dashboard: Update npm packages

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #36565 from jan--f/c-v-py3-flake8

ceph-volume: run flake8 in python3

Merge PR #36351 into master

* refs/pull/36351/head:
qa/tasks/cephfs/nfs: Add tests for cluster config set and reset
doc/cephfs/nfs: Update the doc about 'reset' and 'set' config interfaces
mgr/volumes/nfs: Add interface for adding user defined configuration

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tests: use "-k distro" for all suites (except krbd)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #36437 from yuriw/wip-yuriw-crontab-master

qa/tests: changes related to #36435 and #36436

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #36111 from vosdev/patch-2

doc/mgr/dashboard: Fix haproxy example to use ssl for backends

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Hide table action input field if limit=0

Fixes: https://tracker.ceph.com/issues/46901
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #36456 from sebastian-philipp/orch-type-py3

mgr/orch: _interface.py cleanup: PEP8 and type annotations

Reviewed-by: Joshua Schmid <jschmid@suse.de>

ceph-volume: run flake8 in pyhton3

Fixes: https://tracker.ceph.com/issues/46897
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #36431 from rishabh-d-dave/cv-add-tests-for-new-lvm-functions

ceph-volume: add tests for new functions that run LVM commands

Merge pull request #35937 from rishabh-d-dave/cv-rm-container-classes

ceph-volume: remove container classes from api/lvm.py

Merge pull request #36425 from majianpeng/rocksdb-remove-redundant-perfcount

kv/RocksdbStore: remove perfcounter l_rocksdb_txns/l_rocksdb_txns_sync

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #36528 from dmick/wip-ctags

cmake: don't include tags for Python imports, .tox, build/ dirs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36538 from zdover23/wip-docs-45678-fix

doc/cephadm: Remove shell alias

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

cmake: don't include tags for Python imports, .tox, build/ dirs

For things like cephadm, where there is a lot of "from X import Y",
the import tags become cumbersome.  .tox dirs and
python-common/build are just repeats of source files found elsewhere
so result in duplicate tags

Signed-off-by: Dan Mick <dmick@redhat.com>

mgr/dashboard: Update npm packages

Fixes: https://tracker.ceph.com/issues/46888
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge PR #36221 into master

* refs/pull/36221/head:
client: switch lock_guard to scoped_lock
client: remove useless unsafe_sync_write
client: make the root member under the client_lock
client: add mount/initialize states support and convert to RWRef
client: add RWRef support

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36487 into master

* refs/pull/36487/head:
mds: avoid calling g_conf().get_val() in MDSRank::heartbeat_reset()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36522 from kalebskeithley/rgw_rados_h

rgw: remove unnecessary #include "rgw_rados.h"

Merge PR #32731 into master

* refs/pull/32731/head:
mgr/mds_autoscaler: plugin to configure mds instances

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35161 into master

* refs/pull/35161/head:
mgr/mgr_module: metadata is fetched async
mgr/status: metadata is fetched async

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36548 from votdev/fix_colors

mgr/dashboard: correct color names which are not defined

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: correct color names which are not defined

Fix regression introduced by https://github.com/ceph/ceph/pull/35954.
Rename some color names that have been changed by the mentioned PR.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge PR #36547 into master

* refs/pull/36547/head:
client: add on space before the inode info

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #34869 from linuxbox2/wip-reset-stats2

rgw: introduce safe user-reset-stats

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35316 from cbodley/wip-45771

rgw: fix shutdown crash in RGWAsyncReadMDLogEntries

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #35597 from soumyakoduri/http_client_fix

rgw: Empty reqs_change_state queue before unregistered_reqs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #36015 from cbodley/wip-46456

cls/rgw: preserve olh entry's name on last unlink

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #36516 from tchaikov/wip-crimson-parse-conffile

crimson/common: read conf file in async

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #36069 from mkogan1/wip-fix-endpoint-not-connected-r1

rgw: improve scheduler log

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #35792 from thmour/theo_rgw_url_test

rgw: fix double slash (//) killing the gateway

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #36082 from winndows/superfluous_break3

rgw: Remove superfluous breaks

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36084 from winndows/superfluous_break9

rgw: Remove superfluous breaks

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36092 from Huber-ming/dump_RGWLifecycleConfiguration

rgw: dump transitions in RGWLifecycleConfiguration::dump()

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36264 from theanalyst/rgw/bucket-not-impl

rgw: s3: mark bucket encryption as not implemented

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>