common/PriorityCache: Implement Cache Age Binning

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #44537 from pdvian/wip-typo-fix

rgw: Fix typo error for stat_remote_obj

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44496 from idryomov/wip-diff-iterate-parent-fix

librbd: restore diff-iterate include_parent functionality in fast-diff mode

Reviewed-by: Mykola Golub <mgolub@suse.com>

rgw: Fix typo error for stat_remote_obj

Signed-off-by: Prashant D <pdhange@redhat.com>

Merge pull request #44483 from idryomov/wip-diff-iterate-offset-fix

librbd: diff-iterate reports incorrect offsets in fast-diff mode

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #44355 from sseshasa/wip-fix-enhance-mclock-logging

osd:  Enhance mclockscheduler logging and formatted dump information.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44482 from sebastian-philipp/nfs-upgrade-port-conflict

mgr/cephadm: nfs migration: avoid port conflicts

Reviewed-by: Adam King adking@redhat.com

Merge pull request #36308 from linuxbox2/wip-rgwlc-bcktdel

rgwlc:  remove lc entry on bucket delete

Merge pull request #42891 from ofriedma/wip-ofriedma-rgw-qos-finale

rgw: Add rgw rate limiting per user and per bucket

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38774 from linuxbox2/wip-doc-replsupport

rgw: document S3 bucket replication support

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #43404 from rhcs-dashboard/angular-12-upgradee

mgr/dashboard: Update Angular version to 12

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #44088 from ceph/feature-48388-cache

mgr: TTL cache implementation

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>

Merge pull request #44294 from rhcs-dashboard/feature-bluestore-onode

mgr/dashboard: monitoring:Implement BlueStore onode hit/miss counters into the dashboard

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: neha-ojha <NOT@FOUND>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44456 from jtlayton/wip-53765

mount.ceph: fix the handling of new-syntax device names

Reviewed-by: Venky Shankar <vshankar@redhat.com>

osd: Display scheduler specific info when dumping an OpSchedulerItem

Implement logic to dump information relevant to the scheduler type being
employed when dumping details about an OpSchedulerItem. For e.g., the
'priority' field is relevant for the 'wpq' scheduler, but for the
'mclock_scheduler', the 'qos_cost' gives more information during debugging.

A couple of additional fields called 'qos_cost' and 'is_qos_request' are
introduced in OpSchedulerItem class. These are mainly used to facilitate
dumping of relevant information depending on the scheduler type. The
interesting points are when an item is enqueued and dequeued.

For the 'mclock_scheduler', the 'class_id' and the 'qos_cost' fields are
dumped during enqueue and dequeue op respectively. For the 'wpq' scheduler
things remain the same as before.

An additional benefit of this change is to help immediately identify the
type of scheduler being used for a given shard depending on what is dumped
in the debug messages while debugging.

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

osd: Add debug logs and formatted dumps in the mClockScheduler

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

osd: Change description of 'dump_op_pq_state' command to be more generic

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

common: Introduce 'mclock' debug subsystem

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #44410 from tchaikov/wip-cmake-cleanup

cmake: detect linux/blk/zoned support

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44502 from yuvalif/fix-call-to-mrgw

rgw/multisite/test: pass zero (no) ssl port to mrgw.sh

Merge pull request #44495 from Matan-B/wip-matanb-crimson-flavor

qa/suites: Crimson flavor configuration fix

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: chunmei-liu chunmei.liu@intel.com

test/librbd: make diff-iterate clone tests exercise fast-diff mode

The fast-diff feature wasn't propagated to the clone so these tests
were exercising the slow list_snaps path no matter what RBD_FEATURES
value was supplied to ceph_test_librbd.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: restore diff-iterate include_parent functionality in fast-diff mode

Commit 4429ed4f3f4c ("librbd: switch diff iterate API to use new snaps
list dispatch methods") removed the recursive execute() call.  The new
list_snaps method does indeed handle parent diffs internally but it is
not used in fast-diff mode.  Nothing changed there -- we still need to
load the parent object map, calculate parent object_diff_state, etc.

Fixes: https://tracker.ceph.com/issues/53787
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: stash unmodified include_parent value in DiffContext

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc: rgw: document S3 bucket replication support

Support was added at Octopus.

Fixes: https://tracker.ceph.com/issues/48755
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: Add admin ops API for rate limiting

Add admin ops API for rate limiting and some bug fixes

Signed-off-by: Or Friedmann <ofriedma@redhat.com>

Merge pull request #44221 from BenoitKnecht/ceph-mon-device-metrics

mon: Abort device health when device not found

Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>

Merge pull request #43362 from ybwang0211/pubsub_settingng_spell

rgw:word spelling error in rgw/rgw_pubsub.h file

Merge pull request #42115 from sandrobonazzola/patch-2

doc/install: ensure to open ceph-mon firewall ports

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

rgw/multisite/test: passzero (no) ssl port to mrgw.sh

in multisite tests script.
this is to fix an issue from: 1f3fbf218faa4859f02917828dbdb48be41a579a

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #44301 from yuvalif/wip-yuval-cloudevents

rgw/notifications: add cloudevents support to HTTP endpoint

Merge pull request #44278 from fengchunsong/dpdk-tso

msg/async/dpdk: Add an option to enable/disable TSO

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #44198 from wjwithagen/fix-bluestore-get-io-size

common/blkdev: add missing stubs for OSes not Linux

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #44402 from inspur-wyq/wip-pybind-mgr

pybind/mgr: fix typos

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #44055 from ifed01/wip-ifed-cosmetics-for-optimize-rm

test/store_test, os/bluestore: bunch of minor improvements/cleanups

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #43993 from SMIL-Infra/cmake-systemd-cleanup

cmake: cleanups about systemd and install

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #44423 from 5cs/fix-unshare-blob-test

test: fix bluestore unshare blob test case

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #44416 from yuvalif/wip-yuval-rgw-params-in-multisite

rgw/multisite/test: allow passing rgw parameter to multisite script

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44020 from adk3798/host-ls-scale

mgr/orchestrator: add filtering and count option for orch host ls

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

rgw/notifications: add cloudevents support to HTTP endpoint

following the cloudevents HTTP spec:
https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md
and more specifically this aws-s3 spec:
https://github.com/cloudevents/spec/blob/main/cloudevents/adapters/aws-s3.md

Signed-off-by: yuval Lifshitz <ylifshit@redhat.com>

rgw/common: make "get" methods on RGWHTTPArgs const

Signed-off-by: yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #44412 from cyx1231st/wip-crimson-improve-log

crimson: leverage --default-log-level to configure the global log level

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44473 from johnbent/patch-1

README.md: Update README.md to add link to tracker.ceph.com

Merge pull request #43833 from melissa-kun-li/prevent-cephadm-stray-daemon-tcmurunner

mgr/cephadm: fix tcmu-runner cephadm_stray_daemon

qa/suites: Fix crimson flavor configuration

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge PR #44054 into master

* refs/pull/44054/head:
doc/rados/operations: document pg_num_max
mgr: set max of 32 pgs for .mgr pool
mgr/dashboard: expect pg_num_max property for pools
mon/OSDMonitor: add option --pg-num_max arg for create pool
mon/OSDMonitor: disallow setting pg_num < min or > max
mgr/pg_autoscaler: apply pg_num_max
mon: add pg_num_max pool property

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #44157 from CongMinYin/pwl-small-fixes

librbd/cache/pwl: code cleanup and some minor fixes

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

librbd: diff-iterate reports incorrect offsets in fast-diff mode

If rbd_diff_iterate2() is called on an image offset that doesn't
correspond to an object boundary, the callback is invoked with an
incorrect image offset.  For example, assuming a fully allocated
image, a diff request for 806354944~57344 results in offs=807403520,
len=57344, exists=true invocation, which is ahead by 1048576 bytes.
This occurs only in fast-diff mode, for a diff request on an image
with the fast-diff feature disabled or if whole_object parameter is
set to false the invocation is correct.

This bug goes back to the introduction of fast-diff mode in commit
6d5b969d4206 ("librbd: add diff_iterate2 to API").

Fixes: https://tracker.ceph.com/issues/53784
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mgr/cephadm: nfs migration: avoid port conflicts

The nfs upgrade renames the nfs spec from `nfs.ganesha-{service_id}`
to `nfs.{service_id}`. Previously we used the orphan-daemon check
to remove the old `nfs.ganesha-{service_id}` daemons. This does not work
as sometimes serve() tries to deploy the new daemons before cleaning up the
old daemons. This results in a port conflict breaking the upgrade.

Fixes: https://tracker.ceph.com/issues/53424
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

librbd/cache/pwl: merge multiple output "<<" calls

Follow the principle of fewer "<<" calls and higher performance.
Merge multiple output "<<" calls.

Signed-off-by: Yin Congmin <congmin.yin@intel.com>

Merge pull request #44475 from athanatos/sjust/wip-53783

crimson/os/seastore/cache: fix debug macro usage

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: fix debug macro usage

Introduced via conflict between 277e57 and 632916.

Fixes: https://tracker.ceph.com/issues/53783
Signed-off-by: Samuel Just <sjust@redhat.com>

README.md: Update README.md to add link to tracker.ceph.com

I searched the existing documentation for a link to the tracker and had a hard time finding it. Other folks like myself might like it so prominently displayed.
Added a link to https://tracker.ceph.com/projects/ceph as https://tracker.ceph.com/ is basically blank and not totally intuitive how to get to issues.

Signed-off-by: John Bent <john.bent@seagate.com>

Merge pull request #44429 from cyx1231st/wip-crimson-improve-log-2

crimson/os/seastore: differentiate loggers in seastore

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44420 from aisakaki/lru-cache

crimson/os/seastore/cache: add lru

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #44474 from ljflores/wip-doc-integration-tests

doc/.../testing_integration_tests: fix formatting of teuthology-suite option

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/.../testing_integration_tests: fix formatting of teuthology-suite option

The `--subset` teuthology-suite option should be formatted.

Signed-off-by: Laura Flores <lflores@redhat.com>

rgw: Add rgw rate limiting per user and per bucket

Add rgw rate limiting per user and per bucket

Signed-off-by: Or Friedmann <ofriedma@redhat.com>

RGW - Zipper add attr merge for User

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #39192 from linuxbox2/wip-lc-notify

rgwlc:  optionally support notifications on object expiration

mgr/dashboard: Update Angular version to 12

A full changelog can be seen here: https://blog.angular.io/angular-v12-is-now-available-32ed51fbfd49

For us, the most I had to do is to take care of the min-max validation
and a small CSS change regarding the math()

Fixes: https://tracker.ceph.com/issues/53049
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #44341 from sebastian-philipp/cleanup-snmp

mgr/cephadm: Cleanup snmp

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #44000 from nmshelke/exception-handling-48473

mgr/stats: exception handling for ceph fs perf stats command

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44291 from vshankar/wip-ignore-invalid-client-op

mds: ignore unknown client op when tracking op latency

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44279 from jtlayton/wip-50235

cephfs-shell: add the ability to mount a named filesystem

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44158 from adk3798/agent-failed

mgr/cephadm: don't include agents in CEPHADM_FAILED_DAEMON

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44388 from rhcs-dashboard/02-host-inventory-fix

mgr/dashboard: fix orchestrator/02-hosts-inventory.e2e failure

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #44193 from adk3798/agent-upgrade

mgr/cephadm: speed up upgrade when using agent

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44266 from adk3798/http-reconfig

mgr/cephadm: reconfig agents over http

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

python-common: move test_valid_snmp_gateway_spec from mgr/cephadm

We have to validate to_json() now as well, as we have spcial enums.
Otherwiese we might end up with !!python... representations.

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

python-common: move test_invalid_snmp_gateway_spec from mgr/cephadm

Let's keep the tests in the same package where the class is defined.

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: SNMP: don't write urls manually

this just broken for non-trivial urls. Don't be a bad example

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: SNMP: Don't write default values into the store

Enable us to chage defaults in the future

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: SNMP: use of python3 enums

Little reason to duplicate things ourselves

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44367 from sebastian-philipp/osd-count-fix-again

mgr/cephadm: Fix count for OSDs with OSD specs

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #44374 from fmount/ingress_vip

mgr/cephadm: spec.virtual_ip  param should be used by the ingress daemon

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc/mgr: Add cli api documentation

Signed-off-by: Waad AlKhoury <walkhour@redhat.com>

mgr/cli Redo  cli api mgr module.

Signed-off-by: Waad AlKhoury <walkhour@redhat.com>
Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr: TTLCache basic implementation

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
Fixes: https://tracker.ceph.com/issues/48388

doc/mgr: Add cache documentation

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr/cli lightweight get method api call

Fixes: https://tracker.ceph.com/issues/50311
Signed-off-by: Waad AlKhoury <walkhour@redhat.com>

Merge pull request #44442 from ceph/dependabot-npm_and_yarn-src-pybind-mgr-dashboard-frontend-simplebar-angular-2.3.6

mgr/dashboard: bump simplebar-angular from 2.3.0 to 2.3.6 in /src/pybind/mgr/dashboard/frontend

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: monitoring:Implement BlueStore onode hit/miss counters into the dashboard

Provide the details pulled from Bluestore stats in order to display the onode hit/miss counters

Fixes: https://tracker.ceph.com/issues/53577
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #44361 from xxhdx1985126/wip-53612

crimson/os/seastore/cache: add lba nodes' logical begin addresses to extent_info_t

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

qa: test cases for ceph fs perf stats command

Fixes: https://tracker.ceph.com/issues/48473
Signed-off-by: Nikhilkumar Shelke <nshelke@redhat.com>

mgr/stats: exception handling for ceph fs perf stats command

Fixes: https://tracker.ceph.com/issues/48473
Signed-off-by: Nikhilkumar Shelke <nshelke@redhat.com>

Merge pull request #44458 from cyx1231st/wip-seastore-cleanup-with-empty-transactions

crimson/os/seastore: cleanup with empty transactions

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44268 from lxbsz/mds-fix1

mds: set hb to NULL after the memory released

Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/os/seastore: cleanup with empty transactions

* Add logs to identify OSD operation that submits transaction;
* Mark empty transactions as debug log;
* Misc cleanup;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #44329 from ljflores/wip-telemetry-last-additions

mgr/telemetry: add heap stats and pool application to the perf channel

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #43716 from kamoltat/wip-ksirivad-autoscale-global-flag

mgr/autoscaler: Introduce noautoscale flag

Reviewed-by: Vikhyat Umrao <vikhyat@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

rgwlc: format event.eventName with to_event_string(...)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: add integration tests for lc triggered bucket notifications

Signed-off-by: yuval Lifshitz <ylifshit@redhat.com>

rgwlc: temporary logic to compensate for rgw::sal::Bucket::get_owner() always nullptr

(Acked by Daniel.)  Now with less use-after-free.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #44433 from sebastian-schmid/master

docs/start: fixes typo in hardware recommendation docs

rgwlc:  remove lc entry on bucket delete

Buckets with lifecycle policies installed have a state entry that
must also be deleted when the bucket is removed.

Fixes: https://tracker.ceph.com/issues/46728
N.b., should really be generic, not specific to the RADOS store, but
there doesn't seem to be a clean model for implementing generic side
effects in Zipper, currently.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

mount.ceph: fix the handling of new-syntax device names

With the new mount device syntax, the mount helper will end up
prepending "client." to whatever device string is passed down. If you
then go to recreate that mount later from the info in /proc/mounts,
you'll end up getting back an error because it'll try to prepend
"client." again.

There is no reason to send a fully-qualified principal name down to the
kernel since it can only use client.* principals anyway. Fix the mount
helper to track the unqualified name internally and only fully-qualify
it when we're scraping the config for info.

Fixes: https://tracker.ceph.com/issues/53765
Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #43580 from rhcs-dashboard/cephadm_box-docs

cephadm/box: DiD box documentation

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: sebastian-philipp <sewagner@redhat.com>

Merge pull request #43763 from rhcs-dashboard/cephadm_box

cephadm/box: Cephadm Docker in Docker dev box

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #44449 from rhcs-dashboard/api-docs-e2e-test

mgr/dashboard: add test coverage for API docs (SwaggerUI)

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>