osd/PrimaryLogPG: don't populate watchers if replica

If an object has an established watch, serving a read from replica
eventually leads to an assert in is_degraded_or_backfilling_object()
called from handle_watch_timeout().

The issue is that after can_serve_replica_read() check is satisfied,
we look up the object context.  If not found, we fetch the object info
and then call populate_obc_watchers() which sees the recorded watch and
treats it as unconnected, arming HandleWatchTimeout on replica.

Fixes: https://tracker.ceph.com/issues/45795
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #35381 from tspmelo/wip-fix-cephfs-directories

mgr/dashboard: Fix CephFS's Directories not showing

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35373 from bk201/wip-45870

mgr/dashboard: remove space after service name in the Hosts List table

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35457 from tchaikov/wip-rpm-unitdir

test/crimson: avoid using variadic future

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: avoid using variadic future

see also 3cee25ba41069e9e9b4855f99be20910759440ca

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35405 from ivancich/wip-fix-truncated-bucket-listing

rgw: bucket list/stats truncates for user w/ >1000 buckets

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #33762 from clwluvw/wildcard-subuser

rgw: Add support wildcard subuser for bucket policy

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #32393 from clwluvw/opa-access-key

rgw: Add subuser to OPA request

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35203 from linuxbox2/wip-lc-perfctr

rgwlc: add lifecycle perf counters

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35073 from cbodley/wip-45382

rgw: add quota enforcement to CopyObj

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Or Friedmann <ofriedma@redhat.com>

Merge pull request #34041 from mdw-at-linuxbox/wip-rgw-swstat-loops-2

rgw: fix loop problem with swift stat on account.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: convert rgw_user_max_buckets to new config options

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: clean up code around buckets listing

This commit adds some const correctness, avoids some string/structure
copying, and reformats some code.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: bucket list/stats truncates for user w/ >1000 buckets

The is_truncated flag was not being handled properly and therefore
limited the output on these radosgw-admin commands to 1000 buckets.

Fixes: https://tracker.ceph.com/issues/45630
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #35380 from sebastian-philipp/cephadm-logs-daemon-id

cephadm: More robust way to deduce the systemd unit

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35411 from sebastian-philipp/adopt-9cd9ad020d93b0b420924fec55da307aff8bd422

qa/cephadm: update adopt corpus

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35372 from mgfritch/orch-service-spec-mypy

python-common: add ServiceSpec type checking

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34275 from pritha-srivastava/wip-rgw-sts-oidc-provider

rgw: adds code for creating and managing oidc provider entities in rgw and for offline validation of OpenID Connect Access and ID Token.

rgw: adding code for policy evaluation for ops
like getbucketversioning, putbucketversioning etc

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: Correcting pool name for oidc provider for older
versions of osd.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: fix for make failure in test folder.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adds code for role session to be used an ARN.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adding new iam ops for iam oidc provider
to correct unittest faliure.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: modifying documentation to include explanation of
open id connect provider related REST APIs and removing
references to token introspection for validating incoming
web token.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adds a commom method to remove prefix(www, http and https)
from a url.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adds code to check for client id and cert thumbprint
while validating incoming web token in AssumeRoleWithWebIdentity.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: Added code for offline OpenId Connect/ OAuth 2.0 tokens
if the token is a JWT.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: Adding jwt-cpp header only library for JWT validation.
https://github.com/Thalhammer/jwt-cpp

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adds REST APIs for OpenID connect providers manipulation.

This commit implements:
1. CreateOpenIDConnectProvider
2. DeleteOpenIDConnectProvider
3. ListOpenIDConnectProviders
4. GetOpenIDConnectProviders

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #35340 from mgfritch/cephadm-srv-id-host

mgr/orch: allow '.' char in hostname

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35413 from sebastian-philipp/upgrade-empty-realm

doc/cephadm: Adopt single-site RGW

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cephadm: More robust way to deduce the systemd unit

Generate a better error message when using wrong args.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35365 from sebastian-philipp/doc-orch-unmanaged

doc/mgr: add docs for unmanaged=True

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35311 from tspmelo/wip-ng-tooltip

mgr/dashboard: Use ng-bootstrap for Tooltip

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

doc/cephadm: Adopt single-site RGW

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/cephadm: update adopt corpus

to 9cd9ad020d93b0b420924fec55da307aff8bd422

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Use ng-bootstrap for Tooltip

Fixes: https://tracker.ceph.com/issues/45754
Signed-off-by: Tiago Melo <tmelo@suse.com>

doc/mgr: add docs for unmanaged=True

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35297 from tspmelo/wip-alert

mgr/dashboard: Use ng-bootstrap for Alert

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35375 from sebastian-philipp/doc-cephadm-volker-error

doc/cephadm: Add alternative error for missing CIDR network

Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: remove space after service name in the Hosts List table

Fixes: https://tracker.ceph.com/issues/45870
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/orch: use keyword rather than positional arg

use keyword `service_id` from the orch cli

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #35263 into master

* refs/pull/35263/head:
test_cephfs_shell: use StringIO instead of BytesIO

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

test_cephfs_shell: use StringIO instead of BytesIO

Code in qa/ uses both StringIO and BytesIO. Let's use StringIO
exclusively (unless necessary) for uniformity. The reason for using
StringIO over BytesIO is that tests mostly need stdout as string than
as bytes and StringIO is used more frequently used in qa/ code at this
point.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #35271 from bk201/refactor-template-cephadm

mgr/cephadm: use jinja2 to render templates

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33646 from ofriedma/wip-mkogan-ofriedma-add-cache-api

rgw: Adding data cache and CDN capabilities

Reviewed-by: Mark Kogan <mkogan@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35018 from sebastian-philipp/cephadm-rm-container

cephadm: Manually remove containers

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/dashboard: Use ng-bootstrap for Alert

Fixes: https://tracker.ceph.com/issues/45751
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35176 from aclamk/wip-prio-cache-name

common: PriorityCache's perf counters can have custom name

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #35299 from tspmelo/wip-ng-popover

mgr/dashboard: Use ng-bootstrap for Popover

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use ng-bootstrap for Popover

Fixes: https://tracker.ceph.com/issues/45753
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35348 from dillaman/wip-45714

rbd-mirror: stop local journal replayer first during shut down

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35339 from dillaman/wip-45803

rbd-mirror: don't hold (stale) copy of local image journal pointer

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Fix CephFS's Directories not showing

This is a regretion introduced in fdd8ec8ca9e38bc0cf9a7b31f232b0322b46782a.

We no longer need to check if a tab is selected to load the content.

In that commit we removed the logic that updated the conditional variable,
but  missed removing its use.

Fixes: https://tracker.ceph.com/issues/45877
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35290 from tspmelo/wip-navs

mgr/dashboard: Migrate Tabs from ngx-bootstrap to ng-bootstrap

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34934 from tspmelo/wip-testbed-inject

mgr/dashboard: Use TestBed.inject instead of TestBed.get

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/cephadm: refactor templating in services

Using Jinja2 to render configs for services:
- Grafana
- Alertmanager
- Prometheus
- NFS
- iSCSI

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/cephadm: add template engine Jinja2

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #35359 from ceph/docs_fix_all_available_devices

doc/cephadm: align to all-available-devices flag

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: Add alternative error for missing CIDR network

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: use keyword rather than positional arg

use keyword `service_id` during the ServiceSpec tests

cephadm/tests/test_cephadm.py:51: in <module>
    class TestCephadm(object):
cephadm/tests/test_cephadm.py:508: in TestCephadm
    ???
../../python-common/ceph/deployment/service_spec.py:606: in __init__
    assert service_type == 'iscsi'
E   AssertionError

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: fix RGWSpec mypy errors

cephadm/module.py:34: note: In module imported here:
cephadm/services/cephadmservice.py: note: In member "config" of class "RgwService":
cephadm/services/cephadmservice.py:167: error: Incompatible types in assignment (expression has type "List[str]", variable has type "str")

Signed-off-by: Michael Fritch <mfritch@suse.com>

python-common: fix pytest failure

523: error: Unsupported operand types for + ("str" and "None")
523: note: Right operand is of type "Optional[str]"

Signed-off-by: Michael Fritch <mfritch@suse.com>

python-common: add ServiceSpec type checking

- add missing type checking to NFSServiceSpec and IscsiServiceSpec
- clean-up type checking for ServiceSpec and RGWSpec

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35279 from badone/wip-py2-fix-osd-scrub-repair.sh

qa/*/osd-scrub-repair.sh: Convert to python3 print syntax

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35363 from tchaikov/wip-qa-cronjob

qa/tests: do not pass "-t 2" to teuthology-suite when testing nautilus

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35326 from idryomov/wip-read-from-replica-ec

Objecter: don't attempt to read from non-primary on EC pools

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #35202 into master

* refs/pull/35202/head:
qa: return a string via getfattr

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge pull request #35342 from shinhwagk/patch-2

cephadm: fix package is not running when enabler is not none

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #35328 into master

* refs/pull/35328/head:
qa/tasks/cephfs/test_scrub.py: use umount_wait to avoid ceph-fuse stuck

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35310 from dillaman/wip-librbd-io-image-dispatch

librbd: fix IO processing deadlock and race conditions in image dispatcher

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35322 from tchaikov/wip-mgr-cephadm

mgr/cephadm: drop py2 compatible bits

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #35288 from ZhenLiu94/lznba

ceph.in: add the missing right parenthesis in the help message

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35337 from rzarzynski/wip-crimson-pidfile-err-handling

global, osd/crimson: improve handling of the crimson-osd instance duplication

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Migrate Tabs from ngx-bootstrap to ng-bootstrap

Using ng-bootstrap for Tabs will allow us to easily implement some new features
like only loading 1 tab at a time (already implemented here) and
saving/restoring last opened tab.

Modified the table component to use a clone of the columns list.
Making changes directly to columns var was causing problem when the table was
loaded a second time.

Fixes: https://tracker.ceph.com/issues/45017
Signed-off-by: Tiago Melo <tmelo@suse.com>

doc/cephadm: align to all-available-devices flag

Signed-off-by: Joshua Schmid <jschmid@suse.de>

librbd: clear pending flush contexts on shut down

The IOs are already dispatched but are not cleared from the
container. This can result in an assertion failure in the destructor.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: dropped ImageRequestWQ temporary code from exclusive-lock dispatch

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: restore missing flush on write-block logic

When creating the new image dispatch layer, the original flush
upon write-block was dropped. This is resulting in some random race
conditions where object IO is still in-flight when the write-block
indicates it's complete.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: drop exclusive-lock image dispatch lock before acquiring lock

This avoids a potential race with the exclusive lock being acquired by
another thread while the image dispatch layer attempts to acquire it.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: exclusive lock image dispatch should not wait on IO when setting lock

IO from later dispatch layers might have caused the need to acquire the lock
(i.e. like an image refresh). In that case, the IO will be blocked waiting for
the exclusive lock to be acquired -- but will deadlock waiting for the IO
to flush.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: track journal replay flush requests to prevent race

If a journal replay flush is in-progress when the ImageReplayer is stopped,
it can race and result in an assertion failure due to two attempted shutdowns
of the same journal replay state machine.

Fixes: https://tracker.ceph.com/issues/45409
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35051 from Devp00l/wip-44620

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

qa/tests: do not pass "-t 2" to teuthology-suite when testing nautilus

this change partially reverts
21dacc823be184718584c5fd3b647e87b3eeba42

see also #34171

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35360 from tchaikov/wip-rpm-cephfs-metrics

ceph.spec.in: claim ownership of include/cephfs/metrics

Reviewed-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: claim ownership of include/cephfs/metrics

otherwise, "osc build" fails like:

[ 5545s] libcephfs-devel-16.0.0-2172.ga59fb9ffac.x86_64.rpm: directories not owned by a package:
[ 5545s]  - /usr/include/cephfs/metrics

this addresses a regression introduced by
c669de12e076bcedf3fe6cfe52377713f06d7ceb

Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "rbd-mirror: wait for events to replay before shut down journal replay"

This reverts commit aeccb0330333aad08afea625325b12e01eff909b.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35352 from tchaikov/wip-vstart-runner-create-cluster-only

qa/tasks/vstart_runner: do not teardown test_path if "create-cluster-…

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/dashboard: Use TestBed.inject instead of TestBed.get

This new method has the same behavior, but is type safe.

Fixes: https://tracker.ceph.com/issues/45326
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35300 from tspmelo/wip-ng-typeahead

mgr/dashboard: Use ng-bootstrap for Typeahead

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35031 from tspmelo/wip-get-current-summary

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35282 from bk201/wip-45735

monitoring: fixing some issues in RBD detail dashboard

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35320 from sebastian-philipp/cephadm-more-type-annotations

cephadm: Add a bunch of type annotations

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35262 from jmolmo/cephadm_shared_folders

cephadm: Make easy manager modules development

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35069 from sebastian-philipp/qa-cephadm-add-rgw-and-iscsi

qa/cephadm: Add RGW

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34860 from jschmid1/host_spec_fixes

python-common: only validate host_pattern if present

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/tasks/vstart_runner: do not teardown test_path if "create-cluster-only"

otherwise we could be removing a "None" directory when tearing down the cluster,
and have following failure:

Exception ignored in: <bound method LocalContext.__del__ of <__main__.LocalContext object at 0x7f99fd4a6cc0>>
Traceback (most recent call last):
  File "../qa/tasks/vstart_runner.py", line 1189, in __del__
    shutil.rmtree(self.teuthology_config['test_path'])
  File "/tmp/tmp.mmM2ugspuR/venv/lib/python3.6/shutil.py", line 477, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/tmp/tmp.mmM2ugspuR/venv/lib/python3.6/shutil.py", line 475, in rmtree
    orig_st = os.lstat(path)
TypeError: lstat: path should be string, bytes or os.PathLike, not NoneType

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #34288 into master

* refs/pull/34288/head:
mds: flag backtrace scrub failures for new files as okay

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35257 from athanatos/sjust/wip-lba-tree

crimson: TransactionManager and BtreeLBAManager initial implementations

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #34719 into master

* refs/pull/34719/head:
ceph-fuse: compatible with libfuse3.5 or higher
cmake: to get the header and library from specified path
libfuse: check the libfuse version from the pkconfig/fuse{3}.pc file

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>