crimson/os/seastore/segment_cleaner: set gc size back to 1MB per cycle

This should reduce the conflicts between gc transactions and io transactions

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/btree: allow backref extents to be loaded from disk without initializing

For the purpose of higher space reclaim perf, we load backref extents by
parallelly invoking Cache::get_extent_by_type, which means loaded backref
extents won't be initialized

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/btree: fix log error

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: record replay_from info for dirty extents and alloc infos in segment header/tail

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: better debug outputs

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/transaction_manager: prepare extents for lba_manager/backref_manager's complete_transaction

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/btree: clean up: add print_detail method for btree leaf nodes

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/segment_cleaner: trim journal and reclaim space with the help of backref

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/transaction_manager: scan backref tree on mount

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/cache: record backref extents placement update in Cache::backref_extents

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: add depth parameter to FixedKVBtree::mapped_space_visitor_t

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/cache: replay alloc info deltas

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/segment_cleaener: flush backrefs when trimming journal

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/cache: add extent alloc info to journal records

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: rename rbm_alloc_delta_t to alloc_delta_t

Devices other than rbm need this delta

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: add backref manager

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: lazy backref tree updates for extent (re)placement

Buffer all backref updates(allocs/releases) in Cache.
Also, organize them into an intrusive set, so later backref query can be done efficiently
Backref query should always be first applied to this set and then to the backref manager

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/backref: prevent backref extents from causing transaction conflicts

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/fixed_kv_btree: generalize btree pins to allow them to hold any kind of value

also add get_type() to PhysicalNodePin

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: add operator '+' to paddr_t

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: add seastore_backref subsys

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/btree: generalize fixed kv btree's max key value

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: add backref nodes

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #45887 from bluikko/patch-3

doc: replace spaces with underscores in config option names

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc: remove spaces at line ends and doubles, fix wrapping

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

doc: replace spaces with underscores in config option names

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #45948 from cyx1231st/wip-seastore-cleaner-work

crimson/os/seastore/segment_cleaner: cleanup segment_info_set_t

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #43413 from esmaeil-mirvakili/bluestore-bufferbloat-mitigation

os/bluestore: Add CoDel to BlueStore for Bufferbloat mitigation

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #45920 from adk3798/stop-mgr-nfs-upgrade-fail

qa/suites/orch/cephadm: stop upgrade tests if failures are seen

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #45931 from rkachach/fix_issue_51671

mgr/cephadm: fixing logging output for rm-cluster operation

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45982 from rkachach/fix_issue_52727

mgr/cephadm: do not add _admin label when no-minimize-config is provided

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45934 from ceph/fix-github_triage-master

.github/pr-triage: reorder milestone step

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #45631 from Rethan/feat-mds-slow-ops

mds: add a perf counter to record slow replies

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45978 from tchaikov/wip-cmake-python3-exact

cmake/modules: use exact version of python3 when finding cython

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44151 from gregsfortytwo/wip-cidr-blocklist

Implement CIDR blocklisting

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #45971 from rishabh-d-dave/cephfs-shell-exclam-marks

qa/cephfs: bug fix and some cleanup for test_cephfs_shell.py

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45977 from tchaikov/wip-cmake-dml-cleanup

cmake: try to find dml only if PMEM is enabled

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/cephfs: cleanup some code in test_cephfs_shell.py

Also delete a comment since it's incorrect.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: don't use exclam mark in tests in test_cephfs_shell.py

Exclamation mark is a special character for bash as well as
cephfs-shell. For bash, it substitutes current command with matching
command from command history and for cephfs-shell it runs the command
as OS-level command and not inside the cephfs-shell.

And evey command executed in tests (say "ls") is run by passing it as a
parameter to cephfs-shell command (that is "cephfs-shell -c <conf> --
ls"). So, exclamation mark, when used in tests, is consumed by bash
instead of cephfs-shell.

To avoid these complications it's best (and even simpler!) to issue the
command meant for bash on bash without going through cephfs-shell.

Fixes: https://tracker.ceph.com/issues/55394
Signed-off-by: Rishabh Dave <ridave@redhat.com>

mgr/cephadm: do not add _admin label when no-minimize-config is provided
Fixes: https://tracker.ceph.com/issues/52727
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #45979 from lxbsz/doc_noatime

doc: cephfs: update with atime not supporting explaination

doc: cephfs: update with atime not supporting explaination

Signed-off-by: Xiubo Li <xiubli@redhat.com>

test: osd: add a /0 cidr test for 32-bit...

...and make the OSDMap handle it.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

test: add a /0 cidr test that blocklists EVERYTHING

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

test: add a 128-bit range blocklist test

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #45975 from adamemerson/wip-mrun-bashism

mrun: Change mrun to use bash

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

cmake/modules: use exact version of python3 when finding cython

* CMakeLists.txt:
    always pass "EXACT" to find_package(Python3).
    because per cmake document, "EXACT" only takes effect when
    <Package>_FIND_VERSION_COUNT is greater than 1, where <Package>
    is "Python3". see also cmake/modules/FindPython/Support.cmake
* cmake/modules/AddCephTest.cmake:
    drop redundant find_package(Python3) calls. since Python3 is
    a mandatory requirement for building Ceph, we only need a
    single call of find_package(Python3..) in the top of the source
    tree. the only possible case to repeat it is to ensure that we
    have the correct version of Python3 used in following CMake
    script. but there is no need to repeat it if we just want to
    ensure that we have a python3 interpretor in place.
* cmake/modules/Distutils.cmake:
    always pass "EXACT" to find_package(Python3).
    we should always pass EXACT to find_package() when finding python3,
    this is a follow-up of e2babdfae8c99f39f99a7c8a8f966299b2e62b19

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

cmake: try to find dml only if PMEM is enabled

as the feature dependent on DML library requires PMEM backend.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

tooling: Change mrun to use bash

Since mrun contains some bashisms, have it use bash explicitly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #45973 from ionutbalutoiu/add-windows-tests

qa/crontab/teuthology-cronjobs: add windows suite

qa/crontab/teuthology-cronjobs: add windows suite

Signed-off-by: Ionut Balutoiu <ibalutoiu@cloudbasesolutions.com>

test: add 32-bit ipv4 and 64-bit ipv6 blocklist testing

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

test: add a 30-bit range mask to ipv6 blocklisting tests

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

test: split up ip- and range-based blocklisting tests

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #44230 from optimistyzy/122_add_dml

blk/pmem: use DML library to offload read/write operations in pmem

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #45872 from idryomov/wip-make-check-enable-rbd-caches

run-make-check.sh: enable RBD persistent caches

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

crimson/os/seastore/segment_cleaner: cleanup SegmentProvider interfaces

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: cleanup segment_info_set_t

* better encapsulated interfaces to SegmentCleaner;
* drop unused device related implementations;
* improve related metrics and logs;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #45609 from optimistyzy/wip_pmem_devdax

blk/pmem: Add the devdax support.

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #45048 from ceph/1710

doc: Add Quincy 17.2.0 release notes

doc/releases/quincy.rst: mention log_to_journald

Details about https://tracker.ceph.com/issues/55383

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc: Typo and indentation fix

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #45933 from ionutbalutoiu/qa-add-rbd-wnbd-tests

qa: add basic Ceph on Windows integration test

doc: No changelog for first release

The diff between origin/pacific..origin/quincy is definitely not what we want listed.  There is no Changelog for the first major release.

Signed-off-by: David Galloway <dgallowa@redhat.com>

doc/releases: additional clarification, release date

- address review comments
- emphasize LevelDB removal
- update release date

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #45945 from vshankar/wip-54374

mgr/snap_schedule: add time zone suffix to snapshot dir name

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45916 from tchaikov/wip-python3

cmake/modules: always use the python3 specified in command line

Merge pull request #45864 from cyx1231st/wip-crimson-refactor-with-device-ut

test/crimson/.../test_transaction_manager: test multiple devices

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #45938 from liu-chunmei/seastore-omap-enlarge

crimson/seastore: enlarge omap_leaf_node size

Reviewed-by: Samuel Just <sjust@redhat.com>

qa: add basic Ceph on Windows integration test

Due to lack of Windows support in the Teuthology, the test case adopts
the following workaround:

* Deploy baremetal machine with `ubuntu_latest.yaml` and
  configure it with libvirt KVM.
* Create a libvirt VM and provision it with Windows Server 2019, using
  the official ISO from Microsoft.
* Configure SSH in the Windows VM, and run the tests remotely via SSH.

The implementation of the test case consists of workunit scripts.

`qa/workunits/windows/test_rbd_wnbd.py` is the main Python script
to test Ceph on Windows basic functionality. This is executed in the
libvirt VM configured with Windows Server 2019.

Co-authored-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Co-authored-by: Daniel Vincze <dvincze@cloudbasesolutions.com>
Signed-off-by: Ionut Balutoiu <ibalutoiu@cloudbasesolutions.com>

Merge pull request #45708 from votdev/issue_55133_improve_exception_handling

mgr/dashboard: Imrove error message of '/api/grafana/validation' API endpoint

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: sunilangadi2 <NOT@FOUND>

Merge pull request #45603 from nmshelke/feature-54472

mgr/volumes: set, get, list and remove metadata of subvolume

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45909 from soumyakoduri/wip-skoduri-dbstore-fixes

rgw/dbstore: Handle prefix/delim while listing objects

mgr/snap_schedule: add time zone suffix to snapshot dir name

Fixes: https://tracker.ceph.com/issues/54374
Signed-off-by: Milind Changire <mchangir@redhat.com>
Signed-off-by: Venky Shankar <vshankar@redhat.com>

run-make-check.sh: enable RBD persistent caches

This was attempted in commit 69a7ed4eab36 ("run-make-check: enable
WITH_RBD_RWL when WITH_PMEM is true") but never completed.  We soon
bumped the requirement on libpmem, so WITH_SYSTEM_PMDK=ON wouldn't
have worked anyway.

Enable the RWL mode conditionally based on WITH_RBD_RWL variable.
Enable the SSD mode unconditionally as it has no special dependencies
and can be built on any architecture.

Fixes: https://tracker.ceph.com/issues/55285
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/encoding/check-generated.sh: show diff if binary reencode check fails

Take bf0b161115aa ("test/encoding/check-generated.sh: show diff if cmp
fails") a bit further.  Suggesting "cmp $tmp1 $tmp2" isn't very helpful
since cmp would report just the mismatch offset.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl: WriteLogCacheEntry constructor must initialize flags

Initializing the individual bit field members leaves the remaining two
bits uninitialized and that garbage state gets persisted.

In general, using bit fields in a structure where the layout actually
matters is not desirable.  Even with a few single bits, such as here,
their order, strictly speaking, is not guaranteed:

    An implementation may allocate any addressable storage unit large
    enough to hold a bit-field. If enough space remains, a bit-field
    that immediately follows another bit-field in a structure shall be
    packed into adjacent bits of the same unit. If insufficient space
    remains, whether a bit-field that does not fit is put into the next
    unit or overlaps adjacent units is implementation-defined. The
    order of allocation of bit-fields within a unit (high-order to
    low-order or low-order to high-order) is implementation-defined.
    The alignment of the addressable storage unit is unspecified.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl: initialize generate_test_instances() objects

... to prevent check-generated.sh failures such as:

**** librbd::cache::pwl::WriteLogPoolRoot test 1 dump_json check failed ****
   ceph-dencoder type librbd::cache::pwl::WriteLogPoolRoot select_test 1 dump_json > /tmp/typ-cAoWrqlHC
   ceph-dencoder type librbd::cache::pwl::WriteLogPoolRoot select_test 1 encode decode dump_json > /tmp/typ-ES5yHpfGL
5c5
<     "flushed_sync_gen": 0,
---
>     "flushed_sync_gen": 255,

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl: fix -Wunused-lambda-capture warnings

Reported by clang on "make check" and "make check arm64" builds.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rgw/dbstore: Handle prefix/delim in Bucket::List op

Given a prefix, fetch only those objects matching the prefix.
In addition, skip the entries with "delim" and instead include
those entries in common_prefixes

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/dbstore: DBObject state initialization

Initialize & store 'state' part of DBObject to be
referenced by callers of 'get_obj_state'.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Add the support to use DML library for PMEM device.

The purpose of this patch is to add the initial support to
offload memory/pmem operations by sync usage through hardware path
in DML library.

Signed-off-by: Ziye Yang <ziye.yang@intel.com>

Merge pull request #45866 from liu-chunmei/crimson-longxattr

crimson: check long attr name

Reviewed-by: Samuel Just <sjust@redhat.com>
Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/seastore: enlarge omap_leaf_node size

to avoid double split when key size around 2K.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #44784 from Gaoweinan/master-doc-InconsistentFormat-gaoweinan

doc:InconsistentFormat

.github/pr-triage: reorder milestone step

In `master` the milestone step exits and causes remaining tasks not to be run. I previously tried with the `continue-on-error` flag, but it didn't work, so let's try putting that steps at the end.

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

mgr/cephadm: fixing logging output for rm-cluster operation
Fixes: https://tracker.ceph.com/issues/51671
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #45860 from adk3798/alertmanager-skip-verify

mgr/cephadm: allow setting insecure_skip_verify for alertmanager

Reviewed-by: Francesco Pantano <fpantano@redhat.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #43879 from rishabh-d-dave/vr-clearer-msg

qa/vstart_runner: print "success" explicitly

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45924 from rishabh-d-dave/qa-cephfs-test_cephfs_shell

qa/cephfs: expand note for test_cephfs_shell.py

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45923 from rishabh-d-dave/doc-fix-note

doc/developer_guide: split a note point into two

Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa/cephfs: expand note for test_cephfs_shell.py

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/vstart_runner: print "success" explicitly

This makes it effortless for new vstart_runner.py users to spot that the
test passed.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #45816 from kotreshhr/clone-failure-status

mgr/volumes: Show clone failure reason in clone status command

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc/developer_guide: split a note point into two

Since both the sentences in the note point aren't strictly related to
each other, it's better to split that note point into two.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/suites/orch/cephadm: stop upgrade tests if failures are seen

Otherwise the tests may run forever. This was already done for
mds upgrade sequence, justadding it in the other two places here

Related to: https://tracker.ceph.com/issues/53939

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #45114 from lxbsz/wip-54362

client: do not release the global snaprealm until unmounting

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45688 from lxbsz/fwd

client: stop forwarding the request when exceeding 256 times

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45107 from lxbsz/wip-54345

mds: reset heartbeat when fetching or committing dentries

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45915 from ljflores/wip-dashboard-cypress-password