osd: remove orphan include after PGLSParentFilter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #28824 from dang/zipper

rgw: Project Zipper part 1

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29621 from xiexingguo/wip-build-push-segv

osd/ReplicatedBackend: check against empty data_included before enabling crc

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #29579 from liewegas/wip-big-vs-bluestore

osd: scrub error on big objects; make bluestore refuse to start on big objects

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29663 from cxytt/bugfix

osd/PrimaryLogPG: Avoid accessing destroyed references in finish_degr…

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29646 from zy751713126/get_pool_name_and_id

pybind/rados: add Ioctx.get_pool_id() and Ioctx.get_pool_name()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29675 from tchaikov/wip-remove-parent-pgls-filter

osd/PrimaryLogPG: remove unused "parent" pgls-filter

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29597 from tchaikov/wip-qa/tasks/cbt

qa/tasks/cbt.py: use "git --depth 1 for" faster clone

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29683 from jan--f/c-v-keep-device-list-as-lists

ceph-volume: don't keep device lists as sets

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge PR #29676 into master

* refs/pull/29676/head:
test/unittest_bluefs: always remove temp bdev file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #29581 into master

* refs/pull/29581/head:
os/bluestore: do not set osd_memory_target default from cgroup limit

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge PR #29577 into master

* refs/pull/29577/head:
os/bluestore/KernelDevice: fix RW_IO_MAX constant
os/bluestore/KernelDevice: print aio error extent in hex

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29578 from theanalyst/rgw-user-policy-urlencode

rgw: url decode PutUserPolicy params

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #29670 from linuxbox2/wip-rgwfile-marker

rgw_file: readdir: do not construct markers w/leading '/'

Merge PR #29422 into master

* refs/pull/29422/head:
qa/tasks/mgr/dashboard/test_health: update schema
doc/rados/operations/monitoring: document muting health alerts
qa/standalone/mon/health-mutes: add tests
doc/rados/operations/health-checks: document MON_DISK_{LOW,CRIT,BIG}
doc/rados/operations/health-checks: document OSD_NO_DOWN_OUT_INTERVAL
doc/rados/operations/health-checks: document AUTH_BAD_CAPS
doc/reados/operations/health-checks: document PG_SLOW_SNAP_TRIMMING
doc/rados/operations/health-checks: document MGR_DOWN
mon/HealthCheck: check mutes based on count, not parsing the summary string
mon/health_checks: associate a count with health_alert_t
mon/HealthMonitor: simplify health alert dump
mon/PGMap: use nice timespan for PG stuck warnings
mon/HealthMonitor: allow muted alert counts to decrease but not increase
mon/PGMap: fix summary form for bluestore health alerts
doc/rados/operations/health-alerts: document BLUESTORE_NO_COMPRESSION
mon/PGMap: fix summary form for POOL_APP_NOT_ENABLED
mon/HealthMonitor: persist summary for non-sticky mutes
mon/HealthMonitor: move get_health_status()
mon/HealthMonitor: automatically clear non-sticky mutes when alert clears
mon/HealthMonitor: add gather_all_health_checks helper
mon/HealthMonitor: add sticky flag to mutes
mon/HealthMonitor: expire mutes based on ttl
mon: apply mutes to health [detail]
mon/HealthMonitor: implement mute and unmount commands
mon/HealthMonitor: maintain list of mutes
mon: refactor/simplify health [detail]
mon/health_checks: format 'health summary' with a colon
mon/health_checks: drop dump_summary_compat

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #29537 into master

* refs/pull/29537/head:
os/bluestore/BlueFS: fix device_migrate_to_* to handle varying alloc sizes
os/bluestore/BlueFS: apply shared_alloc_size to shared device
os/bluestore: whitespace
os/bluestore/BlueFS: add bluefs_shared_alloc_size
os/bluestore/BlueStore.cc: start should be >= _get_ondisk_reserved()

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29686 from tchaikov/wip-osdc-wait-for-osdmap

osdc: should release the rwlock before waiting

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26953 from Exotelis/ceph-dashboard-i18ntool

mgr/dashboard: ceph dashboard i18ntool

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29568 from votdev/ignore_dirs

.gitignore: add more stuff

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume: don't keep device lists as sets

This was introduced by #27754. The explicit device lists were cast to
sets but other parts of the code where not updated accordingly. To avoid
touching all code places, only cast to sets for disjoint test and keep
lists otherwise.

Fixes: https://tracker.ceph.com/issues/41292
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #29684 from jan--f/c-v-batch-functional-check-stderr

ceph-volume: fix batch functional tests, idempotent test must check s…

mgr/dashboard: Daemons Page Tables Test (#29469)

mgr/dashboard: Daemons Page Tables Test

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Logs Page E2E Tests (#29434)

mgr/dashboard: Logs Page E2E Tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29420 from ricardoasmarques/fix-default-builder-is-not-a-function

mgr/dashboard: Fixes 'defaultBuilder' is not a function

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Project Zipper Part 1 - Framework and RGWRadosStore

This is the first part of Project Zipper, the Store Abstraction Layer.
It introduces the basic framework, and wraps RGWRados in RGWRadosStore.
The goal over the next few weeks is to do the same for user, bucket, and
object.  This will make most of the remaining users of RGWRados wrapped
in SAL classes, allowing it to be completely absorbed into the private
RGWRadosStore.  This will also expose all the APIs that need to be
pusheg up to higher layers in the SAL.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

osdc: should release the rwlock before waiting

this addresses a regresssion introduced by 20b1ac6e

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume: fix batch functional tests, idempotent test must check stderr

Fixes: https://tracker.ceph.com/issues/41295
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Improve .gitignore

Ignore some Python related caching dirs.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #29536 from batrick/backport-https

scripts: use https url for redmine

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #28997 from b-ranto/wip-push-dash

Make ceph-dashboard require grafana dashboards

Reviewed-by: Zack Cerza <zcerza@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

test/unittest_bluefs: always remove temp bdev file

we leave files in build directory if the test fails. better off
removing them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd, osdc: drop the unused outdata feature of PGLSFilter.

Before this commit PGLSFilter interface was offering the outdata
parameter in its filter() method:

  filter(..., bufferlist& outdata)

OSD was serializing and appending the bufferlist to response to
CEPH_OSD_OP_PGLS_FILTER and CEPH_OSD_OP_PGNLS_FILTER operations.
At the Objecter's side these extra bits were being parsed and
finally stored in NListContext::extra_info. However, it really
looks this member is not used anywhere.

The commit removes the outdata handling on multiple layers: from
PGLSFilter implementations, through OSD till Objecter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd/PrimaryLogPG: remove unused "parent" pgls-filter

it's implemented using `PGLSParentFilter`, and this filter has never been
used. the only possible user would be `cephfs-data-scan`, but it's using
`PGLSCephFSFilter` which is referened with "cephfs.inode_tag".

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PrimaryLogPG: Avoid accessing destroyed references in finish_degraded_object

As follows:
for (auto i = callbacks_for_degraded_object.begin(); i != callbacks_for_degraded_object.end();) {
    finish_degraded_object((i++)->first);
}

void PrimaryLogPG::finish_degraded_object(const hobject_t oid)
{
  if (callbacks_for_degraded_object.count(oid)) {
    contexts.swap(callbacks_for_degraded_object[oid]);
    callbacks_for_degraded_object.erase(oid);   // Release
  }

  map<hobject_t, snapid_t>::iterator i = objects_blocked_on_degraded_snap.find(
    oid.get_head());  // Access
  ...
}

Fixes: https://tracker.ceph.com/issues/41250
Signed-off-by: Tao Ning <ningtao@sangfor.com.cn>

os/bluestore/KernelDevice: fix RW_IO_MAX constant

This depends on the page size.  See:

https://github.com/torvalds/linux/blob/6e6d05360b80f196ed07061327f03346b204abea/include/linux/fs.h#L2305

https://github.com/torvalds/linux/blob/30d1d92a888d03681b927c76a35181b4eed7071f/tools/virtio/linux/kernel.h#L23

Fixes 4d33114a40d5ae0d541c36175977ca22789a3b88

Fixes: https://tracker.ceph.com/issues/41188
Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/mgr/dashboard/test_health: update schema

Also fix the 'checks' field, which is a list of objects, not strings.  (The
test doesn't notice because it's empty.)

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations/monitoring: document muting health alerts

I think someday the docs for how health alerts work (here) and the
enumeration of all actual alerts should be restructured.  For now this
si the simplest placde to fit this!

Signed-off-by: Sage Weil <sage@redhat.com>t

qa/standalone/mon/health-mutes: add tests

Make sure mute and unmute work.  Make sure stick is sticky. Mkae sure
counts can go down bupt if they go upt hte mute clears.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations/health-checks: document MON_DISK_{LOW,CRIT,BIG}

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations/health-checks: document OSD_NO_DOWN_OUT_INTERVAL

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations/health-checks: document AUTH_BAD_CAPS

Signed-off-by: Sage Weil <sage@redhat.com>

doc/reados/operations/health-checks: document PG_SLOW_SNAP_TRIMMING

The mitigation steps are weak, but it's not clear concrete guidance to
provide.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations/health-checks: document MGR_DOWN

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthCheck: check mutes based on count, not parsing the summary string

This is more explicit and robust, and works with the PG warnings, which
don't conform to the "%d ..." form that the other messages do.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/health_checks: associate a count with health_alert_t

0 means this is a singleton.  Otherwise, we can sum this up, either
via merge() or get_or_add().

We always structure this so the count goes toward zero (more healthy), so
if a value is too low, then we count how much too low it is.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: simplify health alert dump

Use dump() member instead of duplicating!  The only reason we had this
before was because the detail portion was optinoal

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: use nice timespan for PG stuck warnings

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: allow muted alert counts to decrease but not increase

If the summary starts with a digit, parse a count.

If the count goes up, clear the mute.

If the count goes down, update the mute so that we ratchet the threshold
down.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: fix summary form for bluestore health alerts

Count goes first.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations/health-alerts: document BLUESTORE_NO_COMPRESSION

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: fix summary form for POOL_APP_NOT_ENABLED

Count goes first.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: persist summary for non-sticky mutes

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: move get_health_status()

This operates exclusively on HealthMonitor members.  Make public member
private again.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: automatically clear non-sticky mutes when alert clears

If the alert goes away, drop the mute.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: add gather_all_health_checks helper

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: add sticky flag to mutes

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: expire mutes based on ttl

Signed-off-by: Sage Weil <sage@redhat.com>

mon: apply mutes to health [detail]

- de-escalate severity
- mark mutes in structured output
- note mutes in summary text output
- mark mutes in detail text output

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: implement mute and unmount commands

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: maintain list of mutes

Signed-off-by: Sage Weil <sage@redhat.com>

mon: refactor/simplify health [detail]

Get rid of single caller helpers.  Instead, assimilate all the checks
together at once, and have two separate blocks, one for formatted, and
one for plaintext output.  Much easier to follow!

Signed-off-by: Sage Weil <sage@redhat.com>

mon/health_checks: format 'health summary' with a colon

Signed-off-by: Sage Weil <sage@redhat.com>

mon/health_checks: drop dump_summary_compat

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: scrub error when objects are larger than osd_max_object_size

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #29431 into master

* refs/pull/29431/head:
qa: fix malformed suite config

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #28652 into master

* refs/pull/28652/head:
cephfs-shell: Add error message for invalid ls commands

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29554 into master

* refs/pull/29554/head:
cephfs-shell: Fix onecmd TypeError

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29552 into master

* refs/pull/29552/head:
cephfs-shell: Convert paths type from string to bytes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

rgw_file: readdir: do not construct markers w/leading '/'

This case arises when listing the top directory of a bucket, and,
with proper continued enumeration, would generate a non-terminating
loop if a directory contained names which sort lexically before '/'.

Fixes: https://tracker.ceph.com/issues/41252
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #29666 from yuriw/wip-yuriw-crontab-master

qa/tests - upped priority for upgrades on master, otherwise they neve…

qa/tests - upped priority for upgrades on master, otherwise they never lock nodes for testing and fail

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

pybind/rados: add get_id and get_name method in class Ioctx

Signed-off-by: Zheng Yin <zhengyin@cmss.chinamobile.com>

mgr/dashboard: Fix e2e issue in HACKING.rst (#29640)

mgr/dashboard: Fix e2e issue in HACKING.rst

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29570 from rhcs-dashboard/new-bucket-utilities-adaptation

mgr/dashboard: adapt bucket tenant API tests to new behaviour

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29634 from rhcs-dashboard/mgr-module-fixes

mgr/dashboard: fix mgr module API tests

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29612 from tchaikov/wip-crimson-perf-test

crimson/test: add CBT based perf tests

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #29644 from anthonyeleven/patch-3

doc: operations: correct 'comma-delimited'

Reviewed-by: Kefu Chai <kchai@redhat.com>

src/script: add run-cbt.sh

this script will be used by jenkins to drive the CBT based test

also, this test can be used by developer for testing crimson or classic
OSD.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/test: add perf tests for crimson

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/test: add script to convert teuthology task config to cbt config

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: operations: correct 'comma-delimited'

CIDR blocks are comma-separated, not comma-delimited.

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #29118 from cbodley/wip-rgw-metadata-servicification

rgw: metadata refactoring

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge branch 'wip-rgw-metadata-servicification'

 Conflicts:
src/rgw/rgw_auth.cc
src/rgw/rgw_auth_registry.h
src/rgw/rgw_auth_s3.h
src/rgw/rgw_bucket.cc
src/rgw/rgw_bucket.h
src/rgw/rgw_data_sync.h
src/rgw/rgw_frontend.h
src/rgw/rgw_log.h
src/rgw/rgw_main.cc
src/rgw/rgw_rados.cc
src/rgw/rgw_rados.h
src/rgw/rgw_rest_s3.h
src/rgw/rgw_rest_sts.h
src/rgw/rgw_swift_auth.h
src/rgw/rgw_user.cc
src/rgw/rgw_user.h
src/rgw/services/svc_sys_obj_core.h

mgr/dashboard: Fix e2e issue in HACKING.rst

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Daemons Page Tables Test

Selects first daemon from list and checks details table is displayed
Clicks performance counters tab and checks performance counters table is displayed

Fixes: https://tracker.ceph.com/issues/41063
Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>

Merge pull request #29633 from hanfengzhe-hi/Fix-decompression-logprint

rgw:Fix rgw decompression log-print

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: fix mgr module API tests

Signed-off-by: alfonsomthd <almartin@redhat.com>

rgw:Fix rgw decompression log-print

The zlib compression takes effect in RGW。
When getting objects failed because of decompress-failed。“ceph-client.rgw” log printed “Compression failed with exit code......”，it should be “deCompression failed with exit code......”。

Signed-off-by: Han Fengzhe <hanfengzhe@hisilicon.com>

Merge pull request #29587 from yuvalif/wip-yuvali-fix-issue-41169

rgw: don't throw when accept errors are happening on frontend

Merge pull request #29615 from tchaikov/wip-qa/tasks/mgr/dashboard/test_health

qa/tasks/mgr/dashboard/test_health: add missing field for test_full_health

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

qa/tasks/mgr/dashboard/test_health: add missing field for test_full_health

fix regressions introduced by a076260e and d6ff61ed

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/ReplicatedBackend: check against empty data_included before enabling crc

The data_included portion could be empty, e.g., due to there are
too many omap entries and can not be sent in one shot, which as
a result may end up crashing the OSD.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/tasks/mgr/dashboard/test_mgr_module: remove enable/disable test from MgrModuleTelemetryTest

telemetry is always enabled since 2d62d71cd46d791294b3803e034c0995fef08b33

Fixes: https://tracker.ceph.com/issues/41186
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29342 from Jeegn-Chen/wip-scrub-extended-sleep

osd: support osd_scrub_extended_sleep

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #29522 from majianpeng/bluestore-optimization

os/bluestore: deferred IO notify and locking optimization

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29418 from junxfl/test1

OSD: modify n.cookie to op.notify.cookie

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29571 from tchaikov/wip-41177

ceph-objectstore-tool: update-mon-db: do not fail if incmap is missing

Reviewed-by: huangjun <huangjun@xsky.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: RGWUserCtl::PutParams::attrs defaults to nullptr

Signed-off-by: Casey Bodley <cbodley@redhat.com>

vstart: pass "--smp 1 --cpuset $osd" to crimson-osd

for optimal performance when testing multiple crimson-osd in a vstart cluster

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29605 from yaozongyou/remove-unused-marco

common: remove unused _STR and STRINGIFY macro

Reviewed-by: Casey Bodley <cbodley@redhat.com>