osdc: Fix crash on OSD failure

OSDSession::lock should not use lockdep, as we violate its rules in a
place that's safe. Revert it to std::mutex.

Thanks to Or Friedmann <ofriedma@redhat.com> for finding the problem.

Fixes: https://tracker.ceph.com/issues/45878
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #34275 from pritha-srivastava/wip-rgw-sts-oidc-provider

rgw: adds code for creating and managing oidc provider entities in rgw and for offline validation of OpenID Connect Access and ID Token.

rgw: adding code for policy evaluation for ops
like getbucketversioning, putbucketversioning etc

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: Correcting pool name for oidc provider for older
versions of osd.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: fix for make failure in test folder.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adds code for role session to be used an ARN.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adding new iam ops for iam oidc provider
to correct unittest faliure.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: modifying documentation to include explanation of
open id connect provider related REST APIs and removing
references to token introspection for validating incoming
web token.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adds a commom method to remove prefix(www, http and https)
from a url.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adds code to check for client id and cert thumbprint
while validating incoming web token in AssumeRoleWithWebIdentity.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: Added code for offline OpenId Connect/ OAuth 2.0 tokens
if the token is a JWT.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: Adding jwt-cpp header only library for JWT validation.
https://github.com/Thalhammer/jwt-cpp

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adds REST APIs for OpenID connect providers manipulation.

This commit implements:
1. CreateOpenIDConnectProvider
2. DeleteOpenIDConnectProvider
3. ListOpenIDConnectProviders
4. GetOpenIDConnectProviders

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #35340 from mgfritch/cephadm-srv-id-host

mgr/orch: allow '.' char in hostname

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35413 from sebastian-philipp/upgrade-empty-realm

doc/cephadm: Adopt single-site RGW

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35365 from sebastian-philipp/doc-orch-unmanaged

doc/mgr: add docs for unmanaged=True

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35311 from tspmelo/wip-ng-tooltip

mgr/dashboard: Use ng-bootstrap for Tooltip

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

doc/cephadm: Adopt single-site RGW

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Use ng-bootstrap for Tooltip

Fixes: https://tracker.ceph.com/issues/45754
Signed-off-by: Tiago Melo <tmelo@suse.com>

doc/mgr: add docs for unmanaged=True

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35297 from tspmelo/wip-alert

mgr/dashboard: Use ng-bootstrap for Alert

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35375 from sebastian-philipp/doc-cephadm-volker-error

doc/cephadm: Add alternative error for missing CIDR network

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #35263 into master

* refs/pull/35263/head:
test_cephfs_shell: use StringIO instead of BytesIO

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

test_cephfs_shell: use StringIO instead of BytesIO

Code in qa/ uses both StringIO and BytesIO. Let's use StringIO
exclusively (unless necessary) for uniformity. The reason for using
StringIO over BytesIO is that tests mostly need stdout as string than
as bytes and StringIO is used more frequently used in qa/ code at this
point.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #35271 from bk201/refactor-template-cephadm

mgr/cephadm: use jinja2 to render templates

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33646 from ofriedma/wip-mkogan-ofriedma-add-cache-api

rgw: Adding data cache and CDN capabilities

Reviewed-by: Mark Kogan <mkogan@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35018 from sebastian-philipp/cephadm-rm-container

cephadm: Manually remove containers

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/dashboard: Use ng-bootstrap for Alert

Fixes: https://tracker.ceph.com/issues/45751
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35176 from aclamk/wip-prio-cache-name

common: PriorityCache's perf counters can have custom name

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #35299 from tspmelo/wip-ng-popover

mgr/dashboard: Use ng-bootstrap for Popover

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use ng-bootstrap for Popover

Fixes: https://tracker.ceph.com/issues/45753
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35348 from dillaman/wip-45714

rbd-mirror: stop local journal replayer first during shut down

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35339 from dillaman/wip-45803

rbd-mirror: don't hold (stale) copy of local image journal pointer

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35290 from tspmelo/wip-navs

mgr/dashboard: Migrate Tabs from ngx-bootstrap to ng-bootstrap

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34934 from tspmelo/wip-testbed-inject

mgr/dashboard: Use TestBed.inject instead of TestBed.get

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/cephadm: refactor templating in services

Using Jinja2 to render configs for services:
- Grafana
- Alertmanager
- Prometheus
- NFS
- iSCSI

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/cephadm: add template engine Jinja2

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #35359 from ceph/docs_fix_all_available_devices

doc/cephadm: align to all-available-devices flag

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: Add alternative error for missing CIDR network

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35279 from badone/wip-py2-fix-osd-scrub-repair.sh

qa/*/osd-scrub-repair.sh: Convert to python3 print syntax

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35363 from tchaikov/wip-qa-cronjob

qa/tests: do not pass "-t 2" to teuthology-suite when testing nautilus

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35326 from idryomov/wip-read-from-replica-ec

Objecter: don't attempt to read from non-primary on EC pools

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #35202 into master

* refs/pull/35202/head:
qa: return a string via getfattr

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge pull request #35342 from shinhwagk/patch-2

cephadm: fix package is not running when enabler is not none

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #35328 into master

* refs/pull/35328/head:
qa/tasks/cephfs/test_scrub.py: use umount_wait to avoid ceph-fuse stuck

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35310 from dillaman/wip-librbd-io-image-dispatch

librbd: fix IO processing deadlock and race conditions in image dispatcher

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35322 from tchaikov/wip-mgr-cephadm

mgr/cephadm: drop py2 compatible bits

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #35288 from ZhenLiu94/lznba

ceph.in: add the missing right parenthesis in the help message

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35337 from rzarzynski/wip-crimson-pidfile-err-handling

global, osd/crimson: improve handling of the crimson-osd instance duplication

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Migrate Tabs from ngx-bootstrap to ng-bootstrap

Using ng-bootstrap for Tabs will allow us to easily implement some new features
like only loading 1 tab at a time (already implemented here) and
saving/restoring last opened tab.

Modified the table component to use a clone of the columns list.
Making changes directly to columns var was causing problem when the table was
loaded a second time.

Fixes: https://tracker.ceph.com/issues/45017
Signed-off-by: Tiago Melo <tmelo@suse.com>

doc/cephadm: align to all-available-devices flag

Signed-off-by: Joshua Schmid <jschmid@suse.de>

librbd: clear pending flush contexts on shut down

The IOs are already dispatched but are not cleared from the
container. This can result in an assertion failure in the destructor.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: dropped ImageRequestWQ temporary code from exclusive-lock dispatch

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: restore missing flush on write-block logic

When creating the new image dispatch layer, the original flush
upon write-block was dropped. This is resulting in some random race
conditions where object IO is still in-flight when the write-block
indicates it's complete.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: drop exclusive-lock image dispatch lock before acquiring lock

This avoids a potential race with the exclusive lock being acquired by
another thread while the image dispatch layer attempts to acquire it.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: exclusive lock image dispatch should not wait on IO when setting lock

IO from later dispatch layers might have caused the need to acquire the lock
(i.e. like an image refresh). In that case, the IO will be blocked waiting for
the exclusive lock to be acquired -- but will deadlock waiting for the IO
to flush.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: track journal replay flush requests to prevent race

If a journal replay flush is in-progress when the ImageReplayer is stopped,
it can race and result in an assertion failure due to two attempted shutdowns
of the same journal replay state machine.

Fixes: https://tracker.ceph.com/issues/45409
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35051 from Devp00l/wip-44620

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

qa/tests: do not pass "-t 2" to teuthology-suite when testing nautilus

this change partially reverts
21dacc823be184718584c5fd3b647e87b3eeba42

see also #34171

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35360 from tchaikov/wip-rpm-cephfs-metrics

ceph.spec.in: claim ownership of include/cephfs/metrics

Reviewed-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: claim ownership of include/cephfs/metrics

otherwise, "osc build" fails like:

[ 5545s] libcephfs-devel-16.0.0-2172.ga59fb9ffac.x86_64.rpm: directories not owned by a package:
[ 5545s]  - /usr/include/cephfs/metrics

this addresses a regression introduced by
c669de12e076bcedf3fe6cfe52377713f06d7ceb

Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "rbd-mirror: wait for events to replay before shut down journal replay"

This reverts commit aeccb0330333aad08afea625325b12e01eff909b.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35352 from tchaikov/wip-vstart-runner-create-cluster-only

qa/tasks/vstart_runner: do not teardown test_path if "create-cluster-…

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/dashboard: Use TestBed.inject instead of TestBed.get

This new method has the same behavior, but is type safe.

Fixes: https://tracker.ceph.com/issues/45326
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35300 from tspmelo/wip-ng-typeahead

mgr/dashboard: Use ng-bootstrap for Typeahead

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35031 from tspmelo/wip-get-current-summary

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35282 from bk201/wip-45735

monitoring: fixing some issues in RBD detail dashboard

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35320 from sebastian-philipp/cephadm-more-type-annotations

cephadm: Add a bunch of type annotations

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35262 from jmolmo/cephadm_shared_folders

cephadm: Make easy manager modules development

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35069 from sebastian-philipp/qa-cephadm-add-rgw-and-iscsi

qa/cephadm: Add RGW

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34860 from jschmid1/host_spec_fixes

python-common: only validate host_pattern if present

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/tasks/vstart_runner: do not teardown test_path if "create-cluster-only"

otherwise we could be removing a "None" directory when tearing down the cluster,
and have following failure:

Exception ignored in: <bound method LocalContext.__del__ of <__main__.LocalContext object at 0x7f99fd4a6cc0>>
Traceback (most recent call last):
  File "../qa/tasks/vstart_runner.py", line 1189, in __del__
    shutil.rmtree(self.teuthology_config['test_path'])
  File "/tmp/tmp.mmM2ugspuR/venv/lib/python3.6/shutil.py", line 477, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/tmp/tmp.mmM2ugspuR/venv/lib/python3.6/shutil.py", line 475, in rmtree
    orig_st = os.lstat(path)
TypeError: lstat: path should be string, bytes or os.PathLike, not NoneType

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #34288 into master

* refs/pull/34288/head:
mds: flag backtrace scrub failures for new files as okay

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35257 from athanatos/sjust/wip-lba-tree

crimson: TransactionManager and BtreeLBAManager initial implementations

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #34719 into master

* refs/pull/34719/head:
ceph-fuse: compatible with libfuse3.5 or higher
cmake: to get the header and library from specified path
libfuse: check the libfuse version from the pkconfig/fuse{3}.pc file

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #34776 into master

* refs/pull/34776/head:
mgr/volumes: Create subvolume with isolated rados namespace

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #34839 into master

* refs/pull/34839/head:
qa/cephfs: add FUSE module before running mount -t fusectl

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #34942 into master

* refs/pull/34942/head:
tools/cephfs: don't bind to public_addr

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35034 into master

* refs/pull/35034/head:
ceph_fuse: enable fuse_debug when the '-d' option is specified

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35057 into master

* refs/pull/35057/head:
pybind/ceph_volume_client: logger.warn() -> logger.warning()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #35193 into master

* refs/pull/35193/head:
qa/tasks/mds_thrash: s/random.sample/random.choice/

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35234 into master

* refs/pull/35234/head:
mds: cleanup uncommitted fragments before mds goes to active

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/dev/seastore.rst: add a bit of information about lba tree etc

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add transaction_manager

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add lba_manager interface with btree_lba_manager impl

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common: add fixed_kv_node_layout with tests

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/root_block: update to handle relative addresses

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/root_block: fix copy constructor

Otherwise, we don't appropriately initialize root.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: add read clean extents to cache

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: remove unnecessary lambda capture

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache.h: actually call complete_load

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache.h: only retire extent if not already retired and not fresh

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cached_extent: add buffer sharing constructor

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cached_extent.h: fix get_overlap

Signed-off-by: Samuel Just <sjust@redhat.com>

cached_extent: add get_crc32 helper

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: distinguish record and block relative paddrs

Blocks get read independently of the surrounding record,
so paddr's embedded directly in a block need to refer
to other blocks within the same record by a block_relative
addr relative to the block's own offset.  By contrast,
deltas to existing blocks need to use record_relative
addrs relative to the first block of the record.

This patch distinguishes the two kinds of relative paddr
(mainly for debugging purposes) and adapts cache, journal,
etc to use the appropriate types.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os: port abseil::Layout to C++17

and adapt it to our needs

* use C++17 instead of homebrew facilities
* use boost::beast::span for Span

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: import abseil::layout

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/byteorder.h: add _le signed types

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #26004 into master

* refs/pull/26004/head:
mds: forward mds metrics to ceph manager w/ quering interfaces
mds: track per session client metrics
mds: record metrics from all MDSs in MDS rank 0
mds: non-rank based interface for sending message to an mds
mds: inter-mds ping-pong message and type
mgr: introduce query/report types for ceph metadata server
mds: new intra-mds message type for forwarding aggregated metrics
client: new message type for providing client side metrics

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>