qa/crontab: reduce frequency of pacific nightlies

To once a week.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41120 into master

* refs/pull/41120/head:
doc/_ext: ignore desc if it is unavailable
doc/_ext: check "default" for None
doc/_ext: print 0B if option value is 0
doc/cephfs: render options using confval directive

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #41102 from soumyakoduri/set_acl_fix

rgw/acl: set_acl shouldn't overwrite bucket attrs

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41093 from idryomov/wip-rbd-pwl-ssd-boundary

librbd/cache/pwl/ssd/WriteLog: don't crash on split log entries

Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

Merge PR #41105 into master

* refs/pull/41105/head:
qa/workunits/cephadm/test_cephadm: test zap-osds
cephadm: add --zap-osds argument to rm_cluster
cephadm: implement zap-osds --fsid ... command

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

librbd/cache/pwl/ssd/WriteLog: don't crash on split log entries

write_log_entries() will split a log entry at the end of the log, the
remainder is written to the beginning at DATA_RING_BUFFER_OFFSET.  On
the read side aio_read_data_block() doesn't handle this case and just
crashes.  Unless the workload in use is <= 4K, the image is rendered
unusable sooner or later.

Fixes: https://tracker.ceph.com/issues/50589
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #41068 from idryomov/wip-rbd-pwl-ssd-capacity

librbd/cache/pwl/ssd/WriteLog: decrement m_bytes_allocated when retiring

Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

Merge pull request #41042 from tchaikov/wip-rbd-cls-sparse-size

librbd: 32-bit and libc++ portability fixups

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

rgw/acl: set_acl shouldn't overwrite bucket attrs

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

doc/_ext: ignore desc if it is unavailable

there is chance that we don't have desc, desc_long or fmt_desc, in that
case, we should just skip desc before checking its length. so, just use
'if desc' which is able to check for None or empty string.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: check "default" for None

default could be None and we don't reference the default value using
`opt.default`, so this variable is always defined. to avoid printing
a None default value, check for a None value. see
https://jinja.palletsprojects.com/en/2.11.x/templates/#none

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: print 0B if option value is 0

before this change, the rendered Default is "0Ei" if the default size is 0,
would be more readable if it is "0B" or "0".

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/cephfs: render options using confval directive

less repeating, better maintainability this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41085 from idryomov/wip-rbd-nuke-source-spec-file

rbd: --source-spec-file should be --source-spec-path

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #41116 from tchaikov/wip-in_addr_t

common/pick_address: define in_addr_t if it is not defined

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41117 from tchaikov/wip-doc-cephfs-conf

doc/cephfs/mds-config-ref: remove duplicated confval

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

common/pick_address: define in_addr_t if it is not defined

neither mingw not not have in_addr_t defined, see
https://docs.microsoft.com/en-us/windows/win32/api/winsock2/ns-winsock2-in_addr,
so define it if it is not defined.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/cephfs/mds-config-ref: remove duplicated confval

this change addresses the warning of:

/home/jenkins-build/build/workspace/ceph-pr-docs/doc/cephfs/mds-config-ref.rst:2: WARNING: duplicate confval_option description of mds_cache_memory_limit, other instance in cephfs/cache-configuration
/home/jenkins-build/build/workspace/ceph-pr-docs/doc/cephfs/mds-config-ref.rst:2: WARNING: duplicate confval_option description of mds_cache_reservation, other instance in cephfs/cache-configuration

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41043 from ifed01/wip-ifed-fix-use-ratio

osd: compute OSD's space usage ratio via raw space utilization

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #41012 into master

* refs/pull/41012/head:
doc/cephfs/cache-config: use config item macros

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40961 from tchaikov/wip-50456

common/pick_address: filter out loopback addresses

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41071 from tchaikov/wip-cmake-disable-vta

cmake: disable "variable tracking" when building rados python binding

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #41091 into master

* refs/pull/41091/head:
doc/cephfs/nfs: add section about ganesha logs
doc/cephfs/nfs: Replace volume/nfs with nfs
doc/cephfs/nfs: add note about export management with volume/nfs interface only

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #41004 from sseshasa/wip-mclock-max-backfill-misc-fix

osd: Override recovery, backfill and sleep related config options during OSD and mclock scheduler initialization.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>

qa/workunits/cephadm/test_cephadm: test zap-osds

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm: add --zap-osds argument to rm_cluster

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm: implement zap-osds --fsid ... command

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #40967 from dang/wip-dang-zipper-user

RGW Zipper consistent load/store APIs for User

Merge pull request #40946 from ifed01/wip-ifed-dump-pool-statfs

osd: admin cmd to dump pools statfs for a specific OSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41014 from smithfarm/wip-mempool-cacheline-49781

qa: verify the benefits of mempool cacheline optimization

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40998 from cyx1231st/wip-seastore-onode-tree-delete

crimson/onode-staged-tree: implement/validate features to erase values from tree

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/cephfs/nfs: add section about ganesha logs

Fixes: https://tracker.ceph.com/issues/50008
Signed-off-by: Varsha Rao <varao@redhat.com>

doc/cephfs/nfs: Replace volume/nfs with nfs

Recently, nfs related code was moved out of volumes plugin[1]. So using the
name volume/nfs for the interface is not appropriate.

[1] https://github.com/ceph/ceph/pull/40526

Signed-off-by: Varsha Rao <varao@redhat.com>

doc/cephfs/nfs: add note about export management with volume/nfs interface only

Signed-off-by: Varsha Rao <varao@redhat.com>

qa: verify the benefits of mempool cacheline optimization

There already is a test to verify the mempool sharding works, in the sense that
it uses at least half of the variables available to count the number of
allocated objects and their total size. This new test verifies that, with
sharding, object counting is at least twice faster than without sharding. It
also collects cacheline contention data with the perf c2c tool. The manual
analysis of this data shows the optimization gain is indeed related to cacheline
contention.

Fixes: https://tracker.ceph.com/issues/49896
Signed-off-by: Loïc Dachary <loic@dachary.org>

librbd/cache/pwl: include head and tail pointers in STATS

While at it, reduce the number of calls to operator<< and drop
the trailing comma.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl: bump "Waiting for allocation" and "Retiring" dout level

Bump "Waiting for allocation" to 5.

"Retiring" is at 20 for rwl and 1 for ssd.  Bump the latter to 20 as
well.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl: use m_bytes_allocated_cap for both rwl and ssd

Follow rwl mode and use AbstractWriteLog::m_bytes_allocated_cap
instead of m_log_pool_ring_buffer_size specific to ssd.  This fixes
"bytes available" calculation in STATS output.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl/ssd/WriteLog: decrement m_bytes_allocated when retiring

Currently if ssd cache is filled to capacity, all future I/O hangs
indefinitely because even though the cache eventually becomes clean
and retires enough entries to get back under RETIRE_HIGH_WATER, this
isn't communicated to AbstractWriteLog::check_allocation().

Fixes: https://tracker.ceph.com/issues/50560
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl/ssd/WriteLog: fix free()/delete mismatch

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl/ssd/WriteLog: fix indentation

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #40502 from zdover23/wip-doc-podman-version-octopus-note-2021-30-Mar

doc/cephadm: add podman version note to install

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #40072 from wornet-mwo/dashboard--grafana-hostname-corrections

mgr/dashboard: Fixed name clash when hostname similar to another

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: p-se <NOT@FOUND>

RGW Zipper - Permenant User storage API

The load/store/remove APIs for User are needed, so make them consistent
and permenant.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #41040 from tchaikov/wip-doc-mds-conf

doc/cephfs/mds-config-ref: render options using confval directive

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rbd: --source-spec-file should be --source-spec-path

It was renamed during development, get rid of the leftovers.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #41028 from mkogan1/wip-rgw-tracker-50467

rgw: fix segfault related to explicit object manifest handling

Merge pull request #41083 from rhcs-dashboard/fix-50580-master

mgr/dashboard: OSDs placement text is unreadable

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #41082 from tchaikov/wip-50558

os/bluestore: Revert" Optimizing the lock of bluestore writing process"

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #41064 from petrutlucian94/win_at_removedir

include: define AT_REMOVEDIR on Windows

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41038 from tchaikov/wip-cmake-liburing

cmake: pass "CC" using configure when building liburing

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

mgr/dashboard:OSDs placement text is unreadable

While displaying the host pattern in the OSDs placement tab, it gets splited with semi-colons. Also adjusted the column size of Container Image ID and Placement columns.

Fixes: https://tracker.ceph.com/issues/50580
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

os/bluestore: Revert" Optimizing the lock of bluestore writing process"

This reverts commit e80d3f7a891af99ba0d4cfe99e5530016ca867ac.

Fixes: https://tracker.ceph.com/issues/50571
Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/onode-staged-tree: implement erase for perf-staged-tree

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement synthetic tests for value erase

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix ownership issues at node-level

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: allow to insert item to an empty node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: move try_downgrade_root() into fix_index()

Specifically, apply_children_merge() doesn't know whether it will be
retired due to fix_parent_index().

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: don't validate child when it's outdated

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: pass origin_left_addr to apply_children_merge()

rebuild_extent() may change the address of the left child, we need to
pass the origin_left_addr in order to validate if it is correct.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/.../node_extent_manager: replay extents may be dirty

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/onode-staged-tree: cleanup tree_utils.h

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: test fix_index() in internal node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: test merge in internal node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement staged::get_prev_slot()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: test merge in leaf node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement layout-level merge

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: test erase in leaf node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement staged::erase()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix container_t::update_size() to accept negative value

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement staged::is_keys_one()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement node-level erase/merge logic

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement delta encode/decode for erase ops

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: improve logs

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement fast Node::get_name()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix get_pivot_index() for leaf node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: be explicit about node num-key/value invariants

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: cleanup node interfaces

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement states for NodeExtentAccessorT

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement NodeExtentManager::retire_extent()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix log

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: remove implemented TODO

Already implemented in c32c1032de1223169a1223d1b8995856e8763df5

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement tree-level erase interfaces

Define and implement erase related interfaces from BTree, BTree::Cursor,
tree_cursor_t, Node to LeafNode.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: add invalid state to tree_cursor_t

If a key-value pair is erased from the tree, the according tree_cursor_t
is untracked and becomes invalid.

From Cursor point-of-view, it becomes an end Cursor if the internal
cursor is invalidated.

From Value point-of-view, it becomes untracked if the value is erased.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: cleanup replace_track/track_split()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: disable concurrent gc

Fix test/perf hang according to 947b103eed0720bc6276274b1b707b221b895a62

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #40493 from CongMinYin/fix-rwl-fragment

librbd/cache/pwl: Fix pmem cache fragment issue

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl: Fix pmem cache fragment issue

I/O may hang due to pmem cache fragment issue when blocks are diffrent
in size. Call pmdk API(pmemobj_defrag) to solve.

Fixes: https://tracker.ceph.com/issues/49879
Signed-off-by: Yin Congmin <congmin.yin@intel.com>

Merge PR #41072 into master

* refs/pull/41072/head:
doc/releases/nautilus: fix a braino in 14.2.20 release note

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #40924 into master

* refs/pull/40924/head:
mgr/cephadm: check hostname resolution before adding host
cephadm: provide a way to checkhost connection without /etc/hosts passed the shell
doc/cephadm: remove /etc/hosts from list of hostname resoltion methods

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #40593 from ideepika/wip-new-testing-params

qa/config/rados: add dispatch delay testing params

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sam Just <sjust@redhat.com>

Merge PR #40941 into master

* refs/pull/40941/head:
qa/suites/rados/cephadm/smoke-roleless: test client-keyring
qa/tasks/cephadm.py: adjust client.admin key mode; place on all hosts
cephadm: distribute client.admin keyring+conf to label:_admin on bootstrap
doc/cephadm: document the default 'admin' label
mgr/cephadm: 'ceph orch client-keyring ...' commands to manage keyring files
mgr/cephadm: reimplement ceph.conf pushing
mgr/cephadm: use _write_remote_file for ceph.conf
mgr/cephadm: _write_remote_file helper
mgr/cephadm: add placementspec for which hosts get ceph.conf

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge PR #40943 into master

* refs/pull/40943/head:
mgr/cephadm: skip ok-to-stop for mons in upgrade if < 3 mons
mgr/cephadm: don't allow upgrade start with less than 2 mgrs

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #41049 into master

* refs/pull/41049/head:
cephadm: re-assimilate user provided conf after mgr created

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #41055 into master

* refs/pull/41055/head:
cephadm: allow several public networks be matched

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #41059 from tchaikov/wip-crimson-clang-cleanup

crimson: clean up issues spotted by the clang compiler

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cmake: disable "variable tracking" when building rados python binding

to silence the warning like:

/home/jenkins-build/build/workspace/ceph-pull-requests/build/src/pybind/rados/rados.c: In function '__pyx_pymod_exec_rados':
/home/jenkins-build/build/workspace/ceph-pull-requests/build/src/pybind/rados/rados.c:78903:30: note: variable tracking size limit exceeded with '-fvar-tracking-assignments', retrying without
78903 | static CYTHON_SMALL_CODE int __pyx_pymod_exec_rados(PyObject *__pyx_pyinit_module)
      |                              ^~~~~~~~~~~~~~~~~~~~~~
cc: warning: /usr/bin/cc: linker input file unused because linking not done

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not pass ${PYTHON_VERSION} to distutils_add_cython_module()

the python2 support was dropped in
5fc657b40dc7d27e84d4b62b5f37d9c771d74fcd, so no need to pass the python
version anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases/nautilus: fix a braino in 14.2.20 release note

Setting auth_expose_insecure_global_id_reclaim to false disables the
"disconnect right after authentication" behavior, not the warning.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #41063 from tchaikov/wip-rgw-move

rgw/rgw_rest_sts: return local variable w/o std::move()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>