rgw/qa: Add QA suite for Keystone service token

This adds a QA suite  for the service token auth in
RGW. The suite runs a workunit that is a bash script
that spawns a fake Keystone server which is then used
as an auth backend for RGW to test the feature.

A python script is then executed that runs a battery
of tests against RadosGW which talks to the fake Keystone
server running in the background.

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/auth: Cache service tokens separately

This changes so that the service tokens is cached
separately in it's own map and LRU list so that the
cache cannot be poisioned and used to lookup an
expired token.

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/auth: Add assert if allow_expired is not false

This adds an assert if allow_expired is not false where
it must be false and should logically only get there when
it's false.

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/doc: Replace underscores with spaces for config opt

Also fixes a gramatical error in the documentation text
itself.

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/auth: Implement Keystone service token as separate TokenExtractor

This change implements a separate TokenExtractor for the
Keystone service token code instead of extending the
TokenExtractor with another method to implement.

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/auth: Pass allow_expired as value in get_from_keystone

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/doc: Add service token config refs under Keystone

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/doc: Update Keystone doc with service token support

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/auth: Use time_t for new expires and dont static_cast

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/auth: static_cast uint64_t to time_t for set_expires

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/auth: Improve set_expires debug output

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/auth: Deny request if service token does not have accepted roles

The default behavior should be that an invalid service token
makes the request invalid even though we have not checked the
X-Auth-Token yet.

This is not required but should reflect the same behavior as other
keystonemiddleware implemented services have.

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/auth: Set see_also backreferences for service token opts

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/auth: Set see_also for service token behavior config opts

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/auth: Make service token expired token expiration configurable

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/auth: Validate accepted roles on service token user

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/auth: Add service token accepted role config opt

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/auth: Add service token feature behind config

This adds the rgw_keystone_service_token_enabled config
option that determines if service tokens is allowed or not.

The config option defaults to false.

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/auth: Update expires for allowed expired token before caching it

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/auth: Add set_expires to token entry

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/auth: Add service token logic for Keystone

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

rgw/auth: Add get_service_token to TokenExtractor

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

Merge pull request #47766 from cbodley/wip-qa-rgw-tempest-meta

qa/rgw/tempest: blocklist failing tests for removal of container metadata

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #48193 from rhcs-dashboard/fix-stoi

exporter: avoid stoi for empty pid_str

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #47900 from Matan-B/wip-matanb-librados-20

examples/librados: Compile with c++20 on Ubuntu

Merge pull request #48191 from phlogistonjohn/jjm-cephadm-tox-coverage

cephadm: add a tox env for generating a coverage report

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #47494 from rhcs-dashboard/pgs-repaired

mgr/prometheus: expose repaired pgs metrics

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #48190 from ronen-fr/wip-rf-replica-init

osd/scrub: make on_replica_init() idempotent again

Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>

exporter: avoid stoi for empty pid_str

Fixes: https://tracker.ceph.com/issues/57619
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #48121 from rzarzynski/wip-crimson-pgadvmap-critical-from

crimson/osd: initialize PGAdvanceMap::from inside critical section

Reviewed-by: Samuel Just <sjust@redhat.com>

cephadm: add a tox env for generating a coverage report

This "dormant" target is here for developers to run locally and generate
a HTML coverage report to review.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #48131 from adamemerson/wip-no-fno-new-ttp-matching

build: Remove -fno-new-ttp-matchingg flag

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

osd/scrub: make on_replica_init() idempotent again

on_replica_init() might be called twice during replica scrub
initiation. Thus, it was designed to cause no issue if called
an extra time. That was broken when the scrubber-backend code
was introduced.

Fixes: https://tracker.ceph.com/issues/57616
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #48172 from mchangir/doc-add-note-about-cephfs-mirroring-paths

doc/cephfs: add note about cephfs-mirroring paths

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc/cephfs: add note about cephfs-mirroring paths

Signed-off-by: Milind Changire <mchangir@redhat.com>

Merge pull request #48149 from batrick/i57586

tools: allow spaces in dentries for first-damage.sh

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #48057 from athanatos/sjust/wip-io-hang

crimson/osd/client_request: use fresh tracking_events/handle instances on requeue

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd/client_request: use fresh tracking_events/handle instances on requeue

See instance_handle_t explanation in client_request.h

Fixes: https://tracker.ceph.com/issues/57494
Fixes: https://tracker.ceph.com/issues/57495
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #48059 from athanatos/sjust/wip-57506

crimson: fix for pgs stuck in +wait along with some debug improvements

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #48091 from rzarzynski/wip-crimson-pgadvmap-complete-in-order

crimson/osd: PGAdvanceMap ops complete in order

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #48134 from rhcs-dashboard/silence-fix

mgr/dashboard: use service call instead of form component

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

tools: allow spaces in dentries for first-damage.sh

Fixes: https://tracker.ceph.com/issues/57586
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: test first damage with dentry containing spaces

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #45369 from jyha200/wip-samplededup

ceph_dedup_tool: adding sample-dedup

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

mgr/prometheus: expose num objects repaired in pool

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #41855 from p-se/compile-cephadm-1

cephadm: compile cephadm

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #47873 from pdvian/wip-log-last

mon/LogMonitor: Fix log last

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #47852 from Matan-B/wip-matanb-radosmodel

test/osd/RadosModel: Improve error debugging

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #47841 from Matan-B/wip-matanb-rados-striper

SimpleRADOSStriper: Avoid moving bufferlists by using deque in read()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #47774 from vumrao/wip-vumrao-57271

blk/KernelDevice: Modify the rotational and discard check log message

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #48156 from cbodley/wip-rgw-unused-throttles

rgw: remove unused AioThrottles in rgw_op.cc

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #46321 from jmolmo/issue_55605

mgr/rook: fix error when trying to get the list of nfs services

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #46961 from tcoldrick-bb/balancer-weekday

mgr/balancer: Disallow 7 as {begin,end}_weekday and 2400 as {begin,end}_time

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #46091 from pdvian/wip-bluestore-error

os/bluestore: Log error to stderr if fail to decode label

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #43791 from rzarzynski/wip-net-perfctrs-for-encrypted-traffic

auth, msg/async: add perf counters for encrypted traffic

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

examples: Compile with c++20

This change will apply to ubuntu only.
Fixes: https://tracker.ceph.com/issues/57163
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #48166 from rhcs-dashboard/fix-empty-pid-path-case

exporter: don't skip loop if pid path is empty

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

exporter: don't skip loop if pid path is empty

Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Fixes issue when pid file config comes empty from config dump which prevents to add metrics. Also get process metrics only if
pid_path isn't empty.

Merge pull request #47986 from lmgdlmgd/patch-3

osd: set per_pool_stats true when OSD has no PG

Reviewed-by: Igor Fedotov <ifedotov@croit.io>

Merge pull request #48139 from ktdreyer/win-simplify-openssl

win32: skip installing openssl man pages

mgr/dashboard: use service call instead of form component

For creating the silence from the notification sidebar, instead of using
the silence form which will require initializing the whole component on
the landing page, we can just call the prometheus service and pass on
the required data to the service call. This will fix showing the
`Prometheus not configured` error everytime we visit the landing page when
the prometheus is not configured

Fixes: https://tracker.ceph.com/issues/57576
Signed-off-by: Nizamudeen A <nia@redhat.com>

ceph_dedup_tool: fix threshold comparison

If threadhold count is 2, existing code always returns true
even though reference count is 1 actually  because it additionally
adds one via ++.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

ceph_dedup_tool: make SampleDedupWorkerThread::chunk_size const

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_dedup_tool: remove unused SampleDedupWorkerThread::duplicable_chunks

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_dedup_tool: minor code simplifications

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_dedup_tool: remove unimplemented mark_non_dedup method signature

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_dedup_tool: fix log lines to use std::endl, update object skip message

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_dedup_tool: remove unused SampleDedupWorkerThread::do_object_dedup

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_dedup_tool: make SampleDedupGlobal::sampling_rate const

Clarifies why it doesn't need to be protected by a lock.

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_dedup_tool: remove unused SampleDedupGlobal::object_dedup_threshold

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_dedup_tool: fix unnecessarily complicated bool returns

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_dedup_tool: SampleDedupWorkerThread can simply inherit from Thread

SampleDedupWorkerThread didn't actually use any of the facilities on
CrawlerThread and indeed left several of them uninitialized.

Also allows us to remove the CrawlerThread constructor which left
several members uninitialized.

Signed-off-by: Samuel Just <sjust@redhat.com>

qa: remove unused values in deduplication.py

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

qa: add validation stage for deduplication.py

To validate sample-dedup actually works, validate() runs
separated thread from sample-dedup and verifies
two following things.
1. check sample-dedup starts properly.
2. check references of all the chunk objects' in chunk tier
   exists in designated base pool.
This routune repeats for max_valication_cnt times while
sample-dedup is running. If it doesn't raise any fail while the loop,
we can pretend sample-dedup works accurately.
If not, assert() will stop this test.

In case that a reference of chunk object doesn't exist in base pool,
validate() gives a second chance after repairing it (chunk-repair op)
to deal with false-positive reference inconsistency.

Signed-off-by: Sungmin Lee <sung_min.lee@samsung.com>

src/tools/ceph_dedup_tool: fixes to daemonize the process correctly

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

qa: add deduplication test under RGW workload (s3)

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #48157 from zdover23/wip-doc-2022-09-18-rados-network-config-prompt-correction

doc/rados: update prompts in network-config-ref

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: update prompts in network-config-ref

This PR adds unselectable prompts to network-config-
ref.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

rgw: remove unused AioThrottles in rgw_op.cc

throttle creation was moved to RadosStore::get_atomic_writer(),
RadosStore::get_append_writer(), and RadosMultipartUpload::get_writer()

these instances in rgw_op.cc were unused

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47849 from torchiaf/fix/37327

mgr/dashboard: Add details to the modal which displays the `safe-to-d…

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #47958 from rhcs-dashboard/edit-host-labels-fix

mgr/dashboard: fix Expected to find element: `cd-modal .badge but never found it

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

win32: skip installing openssl man pages

This speeds up the build a bit and makes the logs easier to read.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #45148 from guits/revert-ceph-crash-changes

Revert "src/ceph-crash.in: remove unused frame in handler()"

Reviewed-by: Teoman ONAY <tonay@redhat.com>

Merge pull request #47965 from cbodley/wip-rgw-drain-cb-capture-error

Revert "rgw: add coroutine drain callback capture_error()"

Reviewed-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

mgr/dashboard: Add details to the modal which displays the `safe-to-destroy` result

- Add warnings type information in the case of the OSDs are not safe to destroy
- Add info type information in the case of the OSDs are safe to destroy

Fixes: https://tracker.ceph.com/issues/37327
Signed-off-by: Francesco Torchia <francesco.torchia@suse.com>

Merge pull request #47878 from xxhdx1985126/wip-pin_set-cpu

crimson/os/seastore/btree: reduce the avg number of queries of pin sets' parent/child lookup

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

build: Remove -fno-new-ttp-matchingg flag

This was added in the upgrade to C++17. It's no longer needed since
fixing Clang compatibility got rid of non-conforming templates.

It's no longer needed and getting rid of it is a (minor) quality of
life enhancement since it gets rid of a spurious error when using
Clang based build tools (language server, etc.) while compiling with
GCC.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #47090 from lxbsz/wip-56553

client: do not uninline data for read

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #48037 from guits/fix_57085

ceph-volume: fix inventory with device arg

crimson/osd: initialize PGAdvanceMap::from inside critical section

Fixes: https://tracker.ceph.com/issues/57542
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #48117 from guits/fix-cv-func-test-filestore-create

ceph-volume/tests: fix lvm centos8-filestore-create job

Merge pull request #48054 from yuvalif/wip-yuval-fix-56572

tests: remove pubsub tests from multisite

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kalpesh Pandya <kapandya@redhat.com>

Merge pull request #47516 from nmshelke/fix-56632

qa: ignore disk quota exceeded failure in test

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

ceph-volume/tests: fix lvm centos8-filestore-create job

/dev/vdc1 can't be zapped if it still holds an lv mapper.
let's use --destroy in the lvm zap command in order to remove
the held lv mapper before zapping the partition and recreate the partition after.

Fixes: https://tracker.ceph.com/issues/57553
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

crimson/os/seastore/btree: reduce the avg number of queries of pin sets' parent/child lookup

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #47966 from cyx1231st/wip-seastore-move-gcprocess

crimson/os/seastore: move AsyncCleaner::GCProcess to EPM::BackgroundProcess

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #48105 from athanatos/sjust/wip-57530

crimson/.../node.cc: fix Node initialization argument passing

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/.../node.cc: fix Node initialization argument passing

Argument evaluation order is unspecified, clang seems to be choosing
to execute the std::unique_ptr move constructor before the call to
get(), causing the first argument to be passed as null.

https://tracker.ceph.com/issues/57530
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #48002 from nSedrickm/accessibility-documentation

doc/dashboard: update documentation with accessibility development pr…

Merge pull request #46723 from nSedrickm/dashboard-health-accessibility-update

mgr/dashboard: Improve accessibility for dashboard health page

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>