crimson/os: add OnodeBlock

used for representing the an node of onode tree

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add crimson::gtest target

instead re-compiling gtest_seastar.cc, better off creating a static
library and link against it instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: add node implementation for onode tree

* node_t and friends are representations of in-mem and on-disk onode
  block/page/node. it might have different names in different contexts.
  but normally, they are the same thing and are loaded as a logical
  cached extent.
* delta_t holds the mutations against a given node onode. they are
  applied to memory once they are committed to disk as part of the
  log record.

regarding to the name of "simple-fltree", it is a short for simple
Flexible Layout Fixed Tree, which was designed by Samuel Just. My
implementation is a naive implementation of this design. But it
tries to be extensive and should be able to adapt to different policies
for rebalancing nodes when overflow/underflow. But there could be
other variant of flexible layout tree implementations in future,
so i added "simple" prefix.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: extract onode.h out

so the concrete onode manager implementation can include it without
including the onode_manager.h interface as a whole

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: add ObjectStore and OnodeManager stubs

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #33878 from theanalyst/rgw/cleanup/user-policy-append

rgw: append user policies to existing policies

Reviewed-by: Pritha Srivastava <prsivas@redhat.com>
Reviewed-by: Adam Emersen <aemerson@redhat.com>

Merge pull request #33521 from linuxbox2/wip-getset-ver-nosuch

rgw: fail when get/set-bucket-versioning attempted on a non-existent …

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #33511 from theanalyst/rgw/append-gc-fixes

rgw: append obj: prevent tail from being GC'ed

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33492 from prasakri/multisite-sync-check

RGW:Multisite: Verify if the synced object is identical to source

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #35470 into master

* refs/pull/35470/head:
mds: fix compile error with CephContext declaration

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35344 into master

* refs/pull/35344/head:
vstart: Support deployment of ganesha daemon by cephadm with NFS option.
mgr/test_orchestrator: List nfs daemon

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #35298 from tspmelo/wip-ng-progressbar

mgr/dashboard: Use ng-bootstrap for Progressbar

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #33083 from mkogan1/wip-rgw-add-beast-access-logs

rgw: add access log to the beast frontend

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #35379 from votdev/issue_45856_show_host_labels

mgr/dashboard: Show labels in hosts page

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mds: fix compile error with CephContext declaration

SimilarTo: https://github.com/ceph/ceph/pull/35209
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rgw: add access log to the beast frontend

Add to the Beast frontend an access log line similar to CivetWeb.
attempting to adhere as much as possible to the Apache Combined Log
Format.

Fixes: https://tracker.ceph.com/issues/45920
rgw: use beast message for access log
(cherry picked from commit 44ec38933cbe6cc864c56332cc99502d86568fdc)

Co-authored-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #35381 from tspmelo/wip-fix-cephfs-directories

mgr/dashboard: Fix CephFS's Directories not showing

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35373 from bk201/wip-45870

mgr/dashboard: remove space after service name in the Hosts List table

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35457 from tchaikov/wip-rpm-unitdir

test/crimson: avoid using variadic future

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: avoid using variadic future

see also 3cee25ba41069e9e9b4855f99be20910759440ca

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35405 from ivancich/wip-fix-truncated-bucket-listing

rgw: bucket list/stats truncates for user w/ >1000 buckets

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #33762 from clwluvw/wildcard-subuser

rgw: Add support wildcard subuser for bucket policy

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #32393 from clwluvw/opa-access-key

rgw: Add subuser to OPA request

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35203 from linuxbox2/wip-lc-perfctr

rgwlc: add lifecycle perf counters

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35073 from cbodley/wip-45382

rgw: add quota enforcement to CopyObj

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Or Friedmann <ofriedma@redhat.com>

Merge pull request #34041 from mdw-at-linuxbox/wip-rgw-swstat-loops-2

rgw: fix loop problem with swift stat on account.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: convert rgw_user_max_buckets to new config options

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: clean up code around buckets listing

This commit adds some const correctness, avoids some string/structure
copying, and reformats some code.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: bucket list/stats truncates for user w/ >1000 buckets

The is_truncated flag was not being handled properly and therefore
limited the output on these radosgw-admin commands to 1000 buckets.

Fixes: https://tracker.ceph.com/issues/45630
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #35380 from sebastian-philipp/cephadm-logs-daemon-id

cephadm: More robust way to deduce the systemd unit

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35411 from sebastian-philipp/adopt-9cd9ad020d93b0b420924fec55da307aff8bd422

qa/cephadm: update adopt corpus

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35372 from mgfritch/orch-service-spec-mypy

python-common: add ServiceSpec type checking

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34275 from pritha-srivastava/wip-rgw-sts-oidc-provider

rgw: adds code for creating and managing oidc provider entities in rgw and for offline validation of OpenID Connect Access and ID Token.

rgw: adding code for policy evaluation for ops
like getbucketversioning, putbucketversioning etc

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: Correcting pool name for oidc provider for older
versions of osd.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: fix for make failure in test folder.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adds code for role session to be used an ARN.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adding new iam ops for iam oidc provider
to correct unittest faliure.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: modifying documentation to include explanation of
open id connect provider related REST APIs and removing
references to token introspection for validating incoming
web token.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adds a commom method to remove prefix(www, http and https)
from a url.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adds code to check for client id and cert thumbprint
while validating incoming web token in AssumeRoleWithWebIdentity.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: Added code for offline OpenId Connect/ OAuth 2.0 tokens
if the token is a JWT.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: Adding jwt-cpp header only library for JWT validation.
https://github.com/Thalhammer/jwt-cpp

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adds REST APIs for OpenID connect providers manipulation.

This commit implements:
1. CreateOpenIDConnectProvider
2. DeleteOpenIDConnectProvider
3. ListOpenIDConnectProviders
4. GetOpenIDConnectProviders

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #35340 from mgfritch/cephadm-srv-id-host

mgr/orch: allow '.' char in hostname

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35413 from sebastian-philipp/upgrade-empty-realm

doc/cephadm: Adopt single-site RGW

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Use ng-bootstrap for Progressbar

Fixes: https://tracker.ceph.com/issues/45752
Signed-off-by: Tiago Melo <tmelo@suse.com>

cephadm: More robust way to deduce the systemd unit

Generate a better error message when using wrong args.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35365 from sebastian-philipp/doc-orch-unmanaged

doc/mgr: add docs for unmanaged=True

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35311 from tspmelo/wip-ng-tooltip

mgr/dashboard: Use ng-bootstrap for Tooltip

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

vstart: Support deployment of ganesha daemon by cephadm with NFS option.

NFS Ganesha cluster can be created in two ways with vstart
cluster:

1) With test_orchestrator
NFS=1 ../src/vstart.sh
This type of deployment can have multiple ganesha daemons on random port.

2) With Cephadm
NFS=1 ../src/vstart.sh --cephadm
It can deploy only single ganesha daemon with vstart on default ganesha port.

Both can create multiple cephfs exports.

Fixes: https://tracker.ceph.com/issues/45830
Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/test_orchestrator: List nfs daemon

This patch adds dummy data for nfs ganesha daemon and adds nfs to list of
supported service type.

Signed-off-by: Varsha Rao <varao@redhat.com>

doc/cephadm: Adopt single-site RGW

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/cephadm: update adopt corpus

to 9cd9ad020d93b0b420924fec55da307aff8bd422

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Use ng-bootstrap for Tooltip

Fixes: https://tracker.ceph.com/issues/45754
Signed-off-by: Tiago Melo <tmelo@suse.com>

doc/mgr: add docs for unmanaged=True

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35297 from tspmelo/wip-alert

mgr/dashboard: Use ng-bootstrap for Alert

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35375 from sebastian-philipp/doc-cephadm-volker-error

doc/cephadm: Add alternative error for missing CIDR network

Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: remove space after service name in the Hosts List table

Fixes: https://tracker.ceph.com/issues/45870
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/orch: use keyword rather than positional arg

use keyword `service_id` from the orch cli

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #35263 into master

* refs/pull/35263/head:
test_cephfs_shell: use StringIO instead of BytesIO

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

test_cephfs_shell: use StringIO instead of BytesIO

Code in qa/ uses both StringIO and BytesIO. Let's use StringIO
exclusively (unless necessary) for uniformity. The reason for using
StringIO over BytesIO is that tests mostly need stdout as string than
as bytes and StringIO is used more frequently used in qa/ code at this
point.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #35271 from bk201/refactor-template-cephadm

mgr/cephadm: use jinja2 to render templates

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33646 from ofriedma/wip-mkogan-ofriedma-add-cache-api

rgw: Adding data cache and CDN capabilities

Reviewed-by: Mark Kogan <mkogan@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35018 from sebastian-philipp/cephadm-rm-container

cephadm: Manually remove containers

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/dashboard: Use ng-bootstrap for Alert

Fixes: https://tracker.ceph.com/issues/45751
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35176 from aclamk/wip-prio-cache-name

common: PriorityCache's perf counters can have custom name

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #35299 from tspmelo/wip-ng-popover

mgr/dashboard: Use ng-bootstrap for Popover

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Show labels in hosts page

Fixes: https://tracker.ceph.com/issues/45856
Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Use ng-bootstrap for Popover

Fixes: https://tracker.ceph.com/issues/45753
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35348 from dillaman/wip-45714

rbd-mirror: stop local journal replayer first during shut down

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35339 from dillaman/wip-45803

rbd-mirror: don't hold (stale) copy of local image journal pointer

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Fix CephFS's Directories not showing

This is a regretion introduced in fdd8ec8ca9e38bc0cf9a7b31f232b0322b46782a.

We no longer need to check if a tab is selected to load the content.

In that commit we removed the logic that updated the conditional variable,
but  missed removing its use.

Fixes: https://tracker.ceph.com/issues/45877
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35290 from tspmelo/wip-navs

mgr/dashboard: Migrate Tabs from ngx-bootstrap to ng-bootstrap

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34934 from tspmelo/wip-testbed-inject

mgr/dashboard: Use TestBed.inject instead of TestBed.get

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/cephadm: refactor templating in services

Using Jinja2 to render configs for services:
- Grafana
- Alertmanager
- Prometheus
- NFS
- iSCSI

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/cephadm: add template engine Jinja2

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #35359 from ceph/docs_fix_all_available_devices

doc/cephadm: align to all-available-devices flag

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: Add alternative error for missing CIDR network

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: use keyword rather than positional arg

use keyword `service_id` during the ServiceSpec tests

cephadm/tests/test_cephadm.py:51: in <module>
    class TestCephadm(object):
cephadm/tests/test_cephadm.py:508: in TestCephadm
    ???
../../python-common/ceph/deployment/service_spec.py:606: in __init__
    assert service_type == 'iscsi'
E   AssertionError

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: fix RGWSpec mypy errors

cephadm/module.py:34: note: In module imported here:
cephadm/services/cephadmservice.py: note: In member "config" of class "RgwService":
cephadm/services/cephadmservice.py:167: error: Incompatible types in assignment (expression has type "List[str]", variable has type "str")

Signed-off-by: Michael Fritch <mfritch@suse.com>

python-common: fix pytest failure

523: error: Unsupported operand types for + ("str" and "None")
523: note: Right operand is of type "Optional[str]"

Signed-off-by: Michael Fritch <mfritch@suse.com>

python-common: add ServiceSpec type checking

- add missing type checking to NFSServiceSpec and IscsiServiceSpec
- clean-up type checking for ServiceSpec and RGWSpec

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35279 from badone/wip-py2-fix-osd-scrub-repair.sh

qa/*/osd-scrub-repair.sh: Convert to python3 print syntax

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35363 from tchaikov/wip-qa-cronjob

qa/tests: do not pass "-t 2" to teuthology-suite when testing nautilus

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35326 from idryomov/wip-read-from-replica-ec

Objecter: don't attempt to read from non-primary on EC pools

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #35202 into master

* refs/pull/35202/head:
qa: return a string via getfattr

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge pull request #35342 from shinhwagk/patch-2

cephadm: fix package is not running when enabler is not none

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #35328 into master

* refs/pull/35328/head:
qa/tasks/cephfs/test_scrub.py: use umount_wait to avoid ceph-fuse stuck

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35310 from dillaman/wip-librbd-io-image-dispatch

librbd: fix IO processing deadlock and race conditions in image dispatcher

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35322 from tchaikov/wip-mgr-cephadm

mgr/cephadm: drop py2 compatible bits

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #35288 from ZhenLiu94/lznba

ceph.in: add the missing right parenthesis in the help message

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35337 from rzarzynski/wip-crimson-pidfile-err-handling

global, osd/crimson: improve handling of the crimson-osd instance duplication

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Migrate Tabs from ngx-bootstrap to ng-bootstrap

Using ng-bootstrap for Tabs will allow us to easily implement some new features
like only loading 1 tab at a time (already implemented here) and
saving/restoring last opened tab.

Modified the table component to use a clone of the columns list.
Making changes directly to columns var was causing problem when the table was
loaded a second time.

Fixes: https://tracker.ceph.com/issues/45017
Signed-off-by: Tiago Melo <tmelo@suse.com>

doc/cephadm: align to all-available-devices flag

Signed-off-by: Joshua Schmid <jschmid@suse.de>

librbd: clear pending flush contexts on shut down

The IOs are already dispatched but are not cleared from the
container. This can result in an assertion failure in the destructor.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: dropped ImageRequestWQ temporary code from exclusive-lock dispatch

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: restore missing flush on write-block logic

When creating the new image dispatch layer, the original flush
upon write-block was dropped. This is resulting in some random race
conditions where object IO is still in-flight when the write-block
indicates it's complete.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: drop exclusive-lock image dispatch lock before acquiring lock

This avoids a potential race with the exclusive lock being acquired by
another thread while the image dispatch layer attempts to acquire it.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: exclusive lock image dispatch should not wait on IO when setting lock

IO from later dispatch layers might have caused the need to acquire the lock
(i.e. like an image refresh). In that case, the IO will be blocked waiting for
the exclusive lock to be acquired -- but will deadlock waiting for the IO
to flush.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>