os/bluestore: warn on no per-pool omap

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: warning (not error) by default on no per-pool omap

Unconditionally error if the global per_pool_omap was set, though!

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: int64_t for error count

Just to be on the safe side.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: default size of 1 TB for testing

Less ENOSPC from vstart.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: behave if we *do* set PGMETA and PERPOOL flags

pgmeta trumps perpool.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not set both PGMETA_OMAP and PERPOOL_OMAP

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: only generate 1 error per omap_head

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make fsck repair convert to per-pool omap

- error if objects not per-pool
- error if per_pool_omap not set
- convert as we go, and set the flag at the end

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: teach fsck to tolerate per-pool omap

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: ondisk format change to 3 for per-pool omap

Move to ondisk format v3.  This means that per-pool omap keys may exist,
but does not imply that *all* objects use the new form until the
per_pool_omap=1 super key is also set.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: add data/omap breakouts for 'df detail' view

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: separate get_{user,allocated}_bytes() into data and omap variants

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: fix stored_raw calculation

The get_user_bytes() helper is a bit weird because it uses the
raw_used_rate (replication/EC factor) so that it can work *backwards*
from raw usage to normalized user usage.  However, the legacy case that
works from PG stats does not use this factor... and the stored_raw value
(in the JSON output only) was incorrectly passing in a factor of 1.0,
which meant that for legacy mode it was a bogus value.

Fix by calculating stored_raw as stored_normalized * raw_used_rate.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: add in actual omap usage into per-pool stats

This is a minimal change: we aren't separately reporting data vs omap
usage (like we do in 'osd df' output for individual osds).

Signed-off-by: Sage Weil <sage@redhat.com>

osd: report per-pool omap support via store_statfs_t

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: set per_pool_omap key on mkfs

This key indicates that *all* objects put omap in the per-pool prefix and
key format.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: count per-pool omap capable OSDs

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: report omap_allocated per-pool

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add pool prefix to omap keys

Set per-onode flag to indicate whether the object has per-pool keys or
not.  This will allow us to incrementally transition objects later.

Put the new keys under a different prefix.

Signed-off-by: Sage Weil <sage@redhat.com>

kv/KeyValueDB: take key_prefix for estimate_prefix_size()

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix manual omap key manipulation to use Onode::get_omap_key()

Make this code more general and robust.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make omap key helpers Onode methods

This makes it easier for them to depend on Onode (Onode::onode) state.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add Onode::get_omap_prefix() helper

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: change _do_omap_clear() args

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29374 into master

* refs/pull/29374/head:
cephfs-shell: Fix flake8 line too long error
cephfs-shell: Fix flake8 line break after binary operator error
cephfs-shell: Fix flake8 continuation line over-indented for hanging indent error
cephfs-shell: Fix flake8 missing whitespace around arithmetic operator error
cephfs-shell: Fix multiple spaces after ':' flake8 error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #27780 from theanalyst/cls-timeindex-denc

cls: add timeindex types to ceph-dencoder

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29182 from dcasier/mgr-dashboard-ganesha-fix-cephfs-name

mgr/dashboard: ganesha: Specify the name of the filesystem (create_path)

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #29138 from Reperator/dashboard-pw-hash

mgr/dashboard: support setting password hashes

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

cephfs-shell: Fix flake8 line too long error

https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix flake8 line break after binary operator error

Fixes: https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix flake8 continuation line over-indented for hanging indent error

Fixes: https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix flake8 missing whitespace around arithmetic operator error

Fixes: https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix multiple spaces after ':' flake8 error

Fixes: https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #29272 from votdev/issue_40925

mgr/dashboard: Various UI issues related to CephFS

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29216 from rafaelquint/configtest

mgr/dashboard: E2E test to verify Configuration editing functionality

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29324 from adk3798/typepages

mgr/dashboard: Explicitly type page variables

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #29371 from bk201/wip-40971

mgr/dashboard: fix MDS counter chart is not displayed

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #29341 from zhangbingyi1/master

test/librados: avoid residual crush rule after test case execution

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix MDS counter chart is not displayed

Unit of timetamp for performance counters is changed to nanoseconds [1].
Adapt this for CephFS MDS performance counters in backend.

[1]: https://github.com/ceph/ceph/pull/28882

Fixes: https://tracker.ceph.com/issues/40971
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

test/librados: avoid residual crush rule after test case execution
Fixes: http://tracker.ceph.com/issues/40970
Signed-off-by: Bingyi Zhang <zhangbingyi@umcloud.com>

Merge pull request #29348 from myoungwon/fix-wrong-class

osd: fix wrong arguments when dropping refcount

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29345 from majianpeng/bluestore-flush-waiter

os/bluestore: avoid unnecessary notify.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #29126 from xiexingguo/wip-clamp-ratio-to-0

mgr/progress: clamp pg recovery ratio to 0

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28012 from rouming/bootstrap-after-setuid

global/global_init: do first transport connection after setuid()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27985 from ifed01/wip-ifed-slow-omaps

tools/ceph-objectstore-tool: introduce list-slow-omap command.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #29238 into master

* refs/pull/29238/head:
os/bluestore: it's better to erase spanning blob only once

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #29239 into master

* refs/pull/29239/head:
qa/tasks/ceph_manager.py: always use self.logger

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #29305 into master

* refs/pull/29305/head:
mgr: Reduce logging noise when handling commands

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29307 into master

* refs/pull/29307/head:
monmaptool: added --addv option to usage description

Reviewed-by: João Eduardo Luís <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29316 into master

* refs/pull/29316/head:
common,tools: make sure the destination buffer can handle the size of the string
src/tools: initialize variables before the goto statement
src/mount: check before dereference buf
src/crush: check before dereference out2
src/test: s/strcpy/strncpy

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29358 into master

* refs/pull/29358/head:
mgr/telemetry: salt osd ids too

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #29054 from dillaman/wip-40621

mgr/rbd_support: support scheduling long-running background operations

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #29361 from tchaikov/wip-qa-osd-pause

qa: fixes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #28685 into master

* refs/pull/28685/head:
client: more precise CEPH_CLIENT_CAPS_PENDING_CAPSNAP
client: unify kicking cap flushes and kicking snapcap flushes
client: define helper function that sends flushsnap message
client: cleanup tracking of early kicked flushing caps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28793 into master

* refs/pull/28793/head:
qa/cephfs: make run_shell() accept args as str too
qa/vstart_runner.py: ignores when source and destination are same
test_cephfs_shell: add a test for cd with arguments
test_cephfs_shell: add a test for cd with no arguments
test_cephfs_shell: group test methods
test_cephfs_shell: add a new method to return script output
test_cephfs_shell: allow running CephFS shell script
cephfs-shell: cd with no args shouldn't print an error message

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29064 into master

* refs/pull/29064/head:
mds: execute PurgeQueue on_error handler in finisher

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29302 into master

* refs/pull/29302/head:
cephfs: fix a memory leak

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29066 into master

* refs/pull/29066/head:
mds: Move inline functions with loops or switch statements to CInode.cc
mds: Reorganize class members in CInode header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29336 into master

* refs/pull/29336/head:
qa: wait for MDS to come back after removing it

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #29359 from jmunhoz/wip-msite-az-tests

rgw: az: add archive zone tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard/qa: add more fields to report

to be in sync with #28847

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/rest/mgr-restful: whitelist OSDMAP_FLAGS

`rest/test-restful.sh` calls `test_mgr_rest_api.py`, which in turn
calls

```
    ('patch',  '/config/osd', {'pause': True}),
```

and rest module translates it to `ceph osd set key=pause`

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: az: add archive zone tests

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

mgr/telemetry: salt osd ids too

Better to fully obfuscate here.

This has a nice side-effect of assigning entity names that are *globally*
unique across the full telemetry data set, since the salts are unique and
sha1 is (sufficiently) collision-free.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29330 into master

* refs/pull/29330/head:
mgr/telemetry: obscure entity_name with a salt

Reviewed-by: Dan Mick <dmick@redhat.com>

mgr/dashboard: Explicitly type page variables

Explicitly gives a type to page variables.
Helps make better use of Typescript's static type checking.
Undoes mistake from #29070.

Fixes: https://tracker.ceph.com/issues/40961
Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>

mgr: Reduce logging noise when handling commands

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #29237 from adk3798/usertest

mgr/dashboard: RGW user E2E Tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #29192 from ceph/dependabot/npm_and_yarn/src/pybind/mgr/dashboard/frontend/lodash-4.17.13

build(deps): bump lodash from 4.17.11 to 4.17.13 in /src/pybind/mgr/dashboard/frontend

Reviewed-by: Tiago Melo <tmelo@suse.com>

pybind/mgr: don't log exception when cannot find RBD task by id

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd: fix wrong arguments when dropping refcount

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

Merge pull request #29198 from rhcs-dashboard/e2e-ci

mgr/dashboard: run e2e tests against prod build (jenkins job)

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29280 from rhcs-dashboard/bucket-tests

mgr/dashboard: RGW bucket creation when no placement target received

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Various UI issues related to CephFS

* Check if data is accessible, otherwise set grafanaId to undefined.
* Catch errors in 'Clients' tab and stop loading spinner. Display ViewCacheStatus Exception.
* Render the 'Enabled' column using the checkIcon cell template as other lists do.

Fixes: https://tracker.ceph.com/issues/40925
Signed-off-by: Volker Theile <vtheile@suse.com>

os/bluestore: avoid unnecessary notify.

Add parameter count waiting list for Onode::flush to avoid unnecessary
notify.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common,tools: make sure the destination buffer can handle the size of the string

fix klocwork issues:
Width is not specified for 's' conversion specifier.
This can result in an overflow of the buffer
provided in argument 3 of a call to 'sscanf'

Signed-off-by: songweibin <song.weibin@zte.com.cn>

src/tools: initialize variables before the goto statement

CID 172143 (#15 of 15): Branch past initialization (PW.BRANCH_PAST_INITIALIZATION)
1. branch_past_initialization:
2. name_at_decl_position: variable "oid" (declared at line 220)
3. name_at_decl_position: variable "exkeys" (declared at line 241)
4. name_at_decl_position: variable "exvals" (declared at line 242)
5. name_at_decl_position: variable "safe" (declared at line 278)

Signed-off-by: songweibin <song.weibin@zte.com.cn>

src/mount: check before dereference buf

CID 180671 (#1 of 1): Dereference null return value (NULL_RETURNS)

Signed-off-by: songweibin <song.weibin@zte.com.cn>

src/crush: check before dereference out2

CID 174874 (#2 of 2): Dereference after null check (FORWARD_NULL)
30. var_deref_op: Dereference null pointer out2.

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #29264 from rjfd/wip-dashboard-fix-qa-rbd

mgr/dashboard: qa: fix RBD test when matching error strings

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28597 from markhpc/wip-bs-ind-cache

os/bluestore: Refactor Bluestore Caches

mds: execute PurgeQueue on_error handler in finisher

Fix: https://tracker.ceph.com/issues/40695
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge PR #28847 into master

* refs/pull/28847/head:
doc/mgr/telemetry: update
mgr/telemetry: move contact info to an 'ident' channel
mgr/telemetry: accept channel list to 'telemetry show'
mgr/telemetry: always generate new report for 'telemetry show'
mgr/telemetry: add 'device' channel and call out to devicehealth module
mgr/telemetry: add telemetry channel 'device'
mgr/telemetry: add separate channels

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge PR #29130 into master

* refs/pull/29130/head:
os/bluestore: Reduce unnecessary wakeups.
os/bluestore: Reduce unnecessary wakeups.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #29221 into master

* refs/pull/29221/head:
mon/OSDMonitor: do not clobber pending xinfo during boot
osd/OSDMap: add is_dead() helper
mon/OSDMonitor: 'osd down ... --definitely-dead'
osd: send MOSDMarkMeDead when we are marked down
mon/OSDMonitor: implement MOSDMarkMeDead handling
mon/OSDMonitor: set dead_epoch for immediate failure (ECONNREFUSED)
osd/OSDMap: add dead_epoch to xinfo

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #29323 into master

* refs/pull/29323/head:
rocksdb: enable rocksdb_rmrange=true by default

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #29300 into master

* refs/pull/29300/head:
doc: fix youtube channel url
doc: fix urls in posix.rst

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28874 into master

* refs/pull/28874/head:
ceph-mds: dump all info of ceph_file_layout, InodeStoreBase, frag_info_t, inode_t

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29103 into master

* refs/pull/29103/head:
mgr/volumes: add `ceph fs subvolumegroup getpath` command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29166 into master

* refs/pull/29166/head:
mds: Reorganize class members in Capability header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #29149 into master

* refs/pull/29149/head:
cephfs-shell: Fix flake8 E302 blank line error
cephfs-shell: Fix flake8 indentation error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28860 into master

* refs/pull/28860/head:
mds: Move scalar type initializations of CDir data members to its header
mds: Reorganize class members in CDir header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

qa: wait for MDS to come back after removing it

Fixes: http://tracker.ceph.com/issues/40967
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon/OSDMonitor: do not clobber pending xinfo during boot

If we have a pending xinfo change and also process a boot message, we
should not throw out the pending xinfo change.

This triggers when you mark an osd down and it sends both a MOSDMarkMeDead
and a new MOSDBoot message in quick succession: the first message sets
dead_epoch but the boot message processing clobbers it.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: add is_dead() helper

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: 'osd down ... --definitely-dead'

Like 'osd down', but we also set dead_epoch.  Should only be used if we
know the process/host is already completely dead and unresponsive.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #29036 from mikechristie/rbd-nbd-add-nl-resize

rbd-nbd: add nl resize

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #28955 into master

* refs/pull/28955/head:
doc: Update mount CephFS index

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29111 into master

* refs/pull/29111/head:
qa/mgr/progress: Update the test suite for progress module

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: send MOSDMarkMeDead when we are marked down

Signed-off-by: Sage Weil <sage@redhat.com>