Merge PR #29577 into master

* refs/pull/29577/head:
os/bluestore/KernelDevice: fix RW_IO_MAX constant
os/bluestore/KernelDevice: print aio error extent in hex

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29578 from theanalyst/rgw-user-policy-urlencode

rgw: url decode PutUserPolicy params

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #29670 from linuxbox2/wip-rgwfile-marker

rgw_file: readdir: do not construct markers w/leading '/'

Merge PR #29422 into master

* refs/pull/29422/head:
qa/tasks/mgr/dashboard/test_health: update schema
doc/rados/operations/monitoring: document muting health alerts
qa/standalone/mon/health-mutes: add tests
doc/rados/operations/health-checks: document MON_DISK_{LOW,CRIT,BIG}
doc/rados/operations/health-checks: document OSD_NO_DOWN_OUT_INTERVAL
doc/rados/operations/health-checks: document AUTH_BAD_CAPS
doc/reados/operations/health-checks: document PG_SLOW_SNAP_TRIMMING
doc/rados/operations/health-checks: document MGR_DOWN
mon/HealthCheck: check mutes based on count, not parsing the summary string
mon/health_checks: associate a count with health_alert_t
mon/HealthMonitor: simplify health alert dump
mon/PGMap: use nice timespan for PG stuck warnings
mon/HealthMonitor: allow muted alert counts to decrease but not increase
mon/PGMap: fix summary form for bluestore health alerts
doc/rados/operations/health-alerts: document BLUESTORE_NO_COMPRESSION
mon/PGMap: fix summary form for POOL_APP_NOT_ENABLED
mon/HealthMonitor: persist summary for non-sticky mutes
mon/HealthMonitor: move get_health_status()
mon/HealthMonitor: automatically clear non-sticky mutes when alert clears
mon/HealthMonitor: add gather_all_health_checks helper
mon/HealthMonitor: add sticky flag to mutes
mon/HealthMonitor: expire mutes based on ttl
mon: apply mutes to health [detail]
mon/HealthMonitor: implement mute and unmount commands
mon/HealthMonitor: maintain list of mutes
mon: refactor/simplify health [detail]
mon/health_checks: format 'health summary' with a colon
mon/health_checks: drop dump_summary_compat

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #29537 into master

* refs/pull/29537/head:
os/bluestore/BlueFS: fix device_migrate_to_* to handle varying alloc sizes
os/bluestore/BlueFS: apply shared_alloc_size to shared device
os/bluestore: whitespace
os/bluestore/BlueFS: add bluefs_shared_alloc_size
os/bluestore/BlueStore.cc: start should be >= _get_ondisk_reserved()

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29686 from tchaikov/wip-osdc-wait-for-osdmap

osdc: should release the rwlock before waiting

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26953 from Exotelis/ceph-dashboard-i18ntool

mgr/dashboard: ceph dashboard i18ntool

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29568 from votdev/ignore_dirs

.gitignore: add more stuff

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29684 from jan--f/c-v-batch-functional-check-stderr

ceph-volume: fix batch functional tests, idempotent test must check s…

mgr/dashboard: Daemons Page Tables Test (#29469)

mgr/dashboard: Daemons Page Tables Test

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Logs Page E2E Tests (#29434)

mgr/dashboard: Logs Page E2E Tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29420 from ricardoasmarques/fix-default-builder-is-not-a-function

mgr/dashboard: Fixes 'defaultBuilder' is not a function

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

osdc: should release the rwlock before waiting

this addresses a regresssion introduced by 20b1ac6e

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume: fix batch functional tests, idempotent test must check stderr

Fixes: https://tracker.ceph.com/issues/41295
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Improve .gitignore

Ignore some Python related caching dirs.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #29536 from batrick/backport-https

scripts: use https url for redmine

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #28997 from b-ranto/wip-push-dash

Make ceph-dashboard require grafana dashboards

Reviewed-by: Zack Cerza <zcerza@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

os/bluestore/KernelDevice: fix RW_IO_MAX constant

This depends on the page size.  See:

https://github.com/torvalds/linux/blob/6e6d05360b80f196ed07061327f03346b204abea/include/linux/fs.h#L2305

https://github.com/torvalds/linux/blob/30d1d92a888d03681b927c76a35181b4eed7071f/tools/virtio/linux/kernel.h#L23

Fixes 4d33114a40d5ae0d541c36175977ca22789a3b88

Fixes: https://tracker.ceph.com/issues/41188
Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/mgr/dashboard/test_health: update schema

Also fix the 'checks' field, which is a list of objects, not strings.  (The
test doesn't notice because it's empty.)

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations/monitoring: document muting health alerts

I think someday the docs for how health alerts work (here) and the
enumeration of all actual alerts should be restructured.  For now this
si the simplest placde to fit this!

Signed-off-by: Sage Weil <sage@redhat.com>t

qa/standalone/mon/health-mutes: add tests

Make sure mute and unmute work.  Make sure stick is sticky. Mkae sure
counts can go down bupt if they go upt hte mute clears.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations/health-checks: document MON_DISK_{LOW,CRIT,BIG}

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations/health-checks: document OSD_NO_DOWN_OUT_INTERVAL

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations/health-checks: document AUTH_BAD_CAPS

Signed-off-by: Sage Weil <sage@redhat.com>

doc/reados/operations/health-checks: document PG_SLOW_SNAP_TRIMMING

The mitigation steps are weak, but it's not clear concrete guidance to
provide.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations/health-checks: document MGR_DOWN

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthCheck: check mutes based on count, not parsing the summary string

This is more explicit and robust, and works with the PG warnings, which
don't conform to the "%d ..." form that the other messages do.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/health_checks: associate a count with health_alert_t

0 means this is a singleton.  Otherwise, we can sum this up, either
via merge() or get_or_add().

We always structure this so the count goes toward zero (more healthy), so
if a value is too low, then we count how much too low it is.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: simplify health alert dump

Use dump() member instead of duplicating!  The only reason we had this
before was because the detail portion was optinoal

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: use nice timespan for PG stuck warnings

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: allow muted alert counts to decrease but not increase

If the summary starts with a digit, parse a count.

If the count goes up, clear the mute.

If the count goes down, update the mute so that we ratchet the threshold
down.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: fix summary form for bluestore health alerts

Count goes first.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations/health-alerts: document BLUESTORE_NO_COMPRESSION

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: fix summary form for POOL_APP_NOT_ENABLED

Count goes first.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: persist summary for non-sticky mutes

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: move get_health_status()

This operates exclusively on HealthMonitor members.  Make public member
private again.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: automatically clear non-sticky mutes when alert clears

If the alert goes away, drop the mute.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: add gather_all_health_checks helper

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: add sticky flag to mutes

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: expire mutes based on ttl

Signed-off-by: Sage Weil <sage@redhat.com>

mon: apply mutes to health [detail]

- de-escalate severity
- mark mutes in structured output
- note mutes in summary text output
- mark mutes in detail text output

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: implement mute and unmount commands

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: maintain list of mutes

Signed-off-by: Sage Weil <sage@redhat.com>

mon: refactor/simplify health [detail]

Get rid of single caller helpers.  Instead, assimilate all the checks
together at once, and have two separate blocks, one for formatted, and
one for plaintext output.  Much easier to follow!

Signed-off-by: Sage Weil <sage@redhat.com>

mon/health_checks: format 'health summary' with a colon

Signed-off-by: Sage Weil <sage@redhat.com>

mon/health_checks: drop dump_summary_compat

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29431 into master

* refs/pull/29431/head:
qa: fix malformed suite config

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #28652 into master

* refs/pull/28652/head:
cephfs-shell: Add error message for invalid ls commands

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29554 into master

* refs/pull/29554/head:
cephfs-shell: Fix onecmd TypeError

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29552 into master

* refs/pull/29552/head:
cephfs-shell: Convert paths type from string to bytes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

rgw_file: readdir: do not construct markers w/leading '/'

This case arises when listing the top directory of a bucket, and,
with proper continued enumeration, would generate a non-terminating
loop if a directory contained names which sort lexically before '/'.

Fixes: https://tracker.ceph.com/issues/41252
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #29666 from yuriw/wip-yuriw-crontab-master

qa/tests - upped priority for upgrades on master, otherwise they neve…

qa/tests - upped priority for upgrades on master, otherwise they never lock nodes for testing and fail

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

mgr/dashboard: Fix e2e issue in HACKING.rst (#29640)

mgr/dashboard: Fix e2e issue in HACKING.rst

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29570 from rhcs-dashboard/new-bucket-utilities-adaptation

mgr/dashboard: adapt bucket tenant API tests to new behaviour

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29634 from rhcs-dashboard/mgr-module-fixes

mgr/dashboard: fix mgr module API tests

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29612 from tchaikov/wip-crimson-perf-test

crimson/test: add CBT based perf tests

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #29644 from anthonyeleven/patch-3

doc: operations: correct 'comma-delimited'

Reviewed-by: Kefu Chai <kchai@redhat.com>

src/script: add run-cbt.sh

this script will be used by jenkins to drive the CBT based test

also, this test can be used by developer for testing crimson or classic
OSD.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/test: add perf tests for crimson

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/test: add script to convert teuthology task config to cbt config

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: operations: correct 'comma-delimited'

CIDR blocks are comma-separated, not comma-delimited.

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #29118 from cbodley/wip-rgw-metadata-servicification

rgw: metadata refactoring

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge branch 'wip-rgw-metadata-servicification'

 Conflicts:
src/rgw/rgw_auth.cc
src/rgw/rgw_auth_registry.h
src/rgw/rgw_auth_s3.h
src/rgw/rgw_bucket.cc
src/rgw/rgw_bucket.h
src/rgw/rgw_data_sync.h
src/rgw/rgw_frontend.h
src/rgw/rgw_log.h
src/rgw/rgw_main.cc
src/rgw/rgw_rados.cc
src/rgw/rgw_rados.h
src/rgw/rgw_rest_s3.h
src/rgw/rgw_rest_sts.h
src/rgw/rgw_swift_auth.h
src/rgw/rgw_user.cc
src/rgw/rgw_user.h
src/rgw/services/svc_sys_obj_core.h

mgr/dashboard: Fix e2e issue in HACKING.rst

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Daemons Page Tables Test

Selects first daemon from list and checks details table is displayed
Clicks performance counters tab and checks performance counters table is displayed

Fixes: https://tracker.ceph.com/issues/41063
Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>

Merge pull request #29633 from hanfengzhe-hi/Fix-decompression-logprint

rgw:Fix rgw decompression log-print

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: fix mgr module API tests

Signed-off-by: alfonsomthd <almartin@redhat.com>

rgw:Fix rgw decompression log-print

The zlib compression takes effect in RGW。
When getting objects failed because of decompress-failed。“ceph-client.rgw” log printed “Compression failed with exit code......”，it should be “deCompression failed with exit code......”。

Signed-off-by: Han Fengzhe <hanfengzhe@hisilicon.com>

Merge pull request #29587 from yuvalif/wip-yuvali-fix-issue-41169

rgw: don't throw when accept errors are happening on frontend

Merge pull request #29615 from tchaikov/wip-qa/tasks/mgr/dashboard/test_health

qa/tasks/mgr/dashboard/test_health: add missing field for test_full_health

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

qa/tasks/mgr/dashboard/test_health: add missing field for test_full_health

fix regressions introduced by a076260e and d6ff61ed

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr/dashboard/test_mgr_module: remove enable/disable test from MgrModuleTelemetryTest

telemetry is always enabled since 2d62d71cd46d791294b3803e034c0995fef08b33

Fixes: https://tracker.ceph.com/issues/41186
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29342 from Jeegn-Chen/wip-scrub-extended-sleep

osd: support osd_scrub_extended_sleep

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #29522 from majianpeng/bluestore-optimization

os/bluestore: deferred IO notify and locking optimization

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29418 from junxfl/test1

OSD: modify n.cookie to op.notify.cookie

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29571 from tchaikov/wip-41177

ceph-objectstore-tool: update-mon-db: do not fail if incmap is missing

Reviewed-by: huangjun <huangjun@xsky.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: RGWUserCtl::PutParams::attrs defaults to nullptr

Signed-off-by: Casey Bodley <cbodley@redhat.com>

vstart: pass "--smp 1 --cpuset $osd" to crimson-osd

for optimal performance when testing multiple crimson-osd in a vstart cluster

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29605 from yaozongyou/remove-unused-marco

common: remove unused _STR and STRINGIFY macro

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29553 from cbodley/wip-rgw-http-unlock-suspend

rgw: http client drops lock before suspending coroutine

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

common: remove unused _STR and STRINGIFY macro

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #29547 from jan--f/c-v-always-log-to-stdout

ceph-volume: never log to stdout, use stderr instead

Merge pull request #29076 from tchaikov/wip-crimson-rep-op

crimson/osd: implement replicated write

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #29378 from cyx1231st/rfc-seastar-msgr-lossless-master

crimson/net: lossless policy for v2 protocol

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: log if the dest of send_to_osd() is not valid

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not return "void"

to avoid potential confusions.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: write pg info after done with peering

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDRepOp

* add a `RepRequest` operation which is blocked by `ConnectionPipeline`
  and `PGPipeline`. these two pipelines are modeled after their
  counterparts of `ClientRequest`.
* add these two blockers to `PG` and `OSDConnectionPriv` accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: replicate transaction to peers

* handle `MOSDRepOpReply` message in osd, and pass it all the way down
  to `PGBackend`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: implement PeeringListener::on_activate()

see also f7b55ec14415811d750e79838f746a91106ba81e

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: minor logging cleanup

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: throw bad_peer_address when reconnect address doesn't match

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: REPLACING state to resolve racing and retain session

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: handle fault for READY, CONNECTING and ACCEPTING

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: WAIT state and backoff for client

Client goes to WAIT state when it is delayed to reconnect, or wants to
be replaced by a newly established socket.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: SERVER_WAIT state for accepting server

Server wait for peer client close the socket at SERVER_WAIT.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: STANDBY state for lossless server or peer

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: allow REPLACING state wait for protocol exit

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: send AckFrame for lossless policy

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>