cephadm/ingress: make frontend stat bind on localhost

The current configuration of keepalived makes it do
a curl on localhost:9999 in order to check the endpoint is alive.
Given the endpoint only binds on the vip addr, that doesn't work.

Fixes: https://tracker.ceph.com/issues/53807
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #44965 from adk3798/test_cli_timeout

qa/tasks/cephadm_cases: increase timeouts in test_cli.py

Reviewed-by: Michael Fritch mfritch@suse.com

Merge pull request #44971 from joscollin/wip-doc-fix-name-id

doc: mention MDS name and ID format in ceph-mds.rst

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #39315 from shenhang/fetchkey

mds: fetch single dentry instead of complete dirfrag for lookup

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44308 from vshankar/wip-document-mds-locker

doc: document mds locking operation

Merge pull request #44898 from taggelos/patch-1

doc/man/8: Fix spelling in user rename example

Merge pull request #44780 from sosodev/dump-blocked-ops-count

mds, osd: add dump_blocked_ops_count command

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44752 from NitzanMordhai/wip-nitzan-osd-status

pybind/mgr: ceph osd status crash with ZeroDivisionError

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #44588 from kamoltat/wip-ksirivad-disable-progress-by-default

pybind/mgr/progress: disable pg recovery event by default

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #44289 from Vicente-Cheng/fix-outgoing-bl-overflow

msg/async: fix outgoing_bl overflow and reset state_offset

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #44809 from cyx1231st/wip-crimson-cleanup-epm

crimson/os/seastore: allow EPM to make decisions on the general allocation path

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

doc: document mds locking operation

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44849 from rhcs-dashboard/bugfix-cephfs-invalid-dir-permission

mgr/dashboard: Directories Menu Can't Use on Ceph File System Dashboard

Merge pull request #44970 from kotreshhr/fix-clone-uid-gid

mgr/volumes: Fix clone uid/gid mismatch

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc: mention MDS name and ID format in ceph-mds.rst

Signed-off-by: Jos Collin <jcollin@redhat.com>

crimson/os/seastore: misc log improvements

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: initialize segment_info_t correctly when open/empty

Previously, segment_info_t was not initialized for out-of-line segments
when open, including journal_segment_seq and the legacy out_of_line.

And the journal_segment_seq was not reset when empty.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: replace out_of_line by segment_type_t

Leverage segment_seq_t to identify the type of the segment.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: cleanup dependency between epm and journal

OOL segments won't contain deltas and don't need to replay. There is no
need to assign segment sequence from journal.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: cleanup dependency between epm and lba_manager

Move lba_manamger->update_mapping() from epm to transaction manager.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: update_mapping related cleanups

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: allow EPM to make decisions on the general extent allocation path

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: drop the unnecessary backend_type in every cached extent

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/epm: cleanup unnecessary cache dependency when mark extent ool

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: cleanup delayed paddr

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #44968 from cyx1231st/wip-crimson-cleanup-cleaner-1

crimson/os/seastore/segment_cleaner: minor cleanups

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44964 from yaarith/pending-release-notes-perf

PendingReleaseNotes: add a note about perf channel

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44963 from ljflores/wip-peering-state-docs

doc/dev: add section on using the gen_state_diagram.py script

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/dev: add section on using the gen_state_diagram.py script

Commands sourced from Jianshen Liu's blog post on the Ceph peering state machine: https://jianshenliu.com/blog/development/ceph-peering-state-machine/

Also fixed the formatting of "PG info".

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #44765 from guits/guits-fix-cve-cv

ceph-volume: honour osd_dmcrypt_key_size option

Merge pull request #44808 from rhcs-dashboard/support-snmp-service-creation

mgr/dashboard: support snmp-gateway service creation from UI

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: sunilangadi2 <NOT@FOUND>

mgr/volumes: Fix clone uid/gid mismatch

This is the regression caused by commit 18b85c53a.
The 'set_attrs' function sets the uid/gid of the
group to the subvolume if uid/gid is not passed.
The attrs of the clone should match the source
snapshot. Hence, don't use the 'set_attrs'
function to set only the quota attrs for the
clone.

Fixes: https://tracker.ceph.com/issues/54066
Signed-off-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #44953 from aejv/spellingerror

purge_queue.py: fixed typo

Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/os/seastore/segment_cleaner: mark get_next_gc_target() private

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: make init_mark_segment_closed() private

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: drop unused code

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net/socket: add a missing header

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

ceph-volume/activate: load the config from lv tag

When `ceph-volume lvm trigger` is called with an OSD where the tag
`ceph.cluster_name` is not 'ceph', it fails.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

qa/tasks/cephadm_cases: increase timeouts in test_cli.py

These seem to be failing sometimes but in my testing
sometimes these events are happening a few seconds after
we hit the timeout. Trying to see if this makes the tests
more consistent. No need to mark the test as failed
if we report something up in 34 seconds vs 25 especially
when cephadm works on a cyclic daemon refresh.

Signed-off-by: Adam King <adking@redhat.com>

ceph-volume/tests: use centos/stream8 images

Since recent move from CentOS 8 to CentOS Stream 8, let's do the same here.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume/tests: add tests in util/encryption.py

this adds some unit tests in order to cover `luks_format()` and `luks_open()`
in `util/encryption.py`.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume: honour osd_dmcrypt_key_size option

ceph-volume doesn't honour osd_dmcrypt_key_size.
It means the default size is always applied.

It also changes the default value in `get_key_size_from_conf()`

From cryptsetup manpage:

> For XTS mode you can optionally set a key size of 512 bits with the -s option.

Using more than 512bits will end up with the following error message:

```
Key size in XTS mode must be 256 or 512 bits.
```

Fixes: https://tracker.ceph.com/issues/54006
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

PendingReleaseNotes: add a note about perf channel

Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>

Merge pull request #44956 from rhcs-dashboard/appli-fix

mgr/dashboard: change the readFile to readFileSync

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #44939 from soumyakoduri/wip-skoduri-dbstore-tests

rgw/dbstore: Add dbstore-tests to `make check`

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: change the readFile to readFileSync

Apparently the readFile i added in #44934 is async and that's not what
we want. so changing it to the synchronous call that is readFileSync

Fixes: https://tracker.ceph.com/issues/54190
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge PR #42000 into master

* refs/pull/42000/head:
qa: update rhel kclient to setup container tools
qa: stop overriding distro for k-testing
qa: only use RHEL for workload testing
qa: convert fs:workload to use cephadm
qa: split fs begin task
qa/tasks/cephadm: setup CephManager when OSDs are provisioned
qa/tasks/cephadm: setup file system if MDS are provisioned

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

purge_queue.py: fixed typo

Signed-off-by: Judah van der Ster <jevanderster@gmail.com>

Merge pull request #44937 from idryomov/wip-rbd-help-positional-optional

rbd: mark optional positional arguments as such in help output

Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #44492 from xijiacun/master

rgw: Erase old storage class attr when the object is rewrited using r…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard:Directories Menu Can't Use on Ceph File System Dashboard

Added exception handling to opendir() in cephfs.py for directories with no execute permission.

Fixes: https://tracker.ceph.com/issues/51611
Signed-off-by: Sarthak0702 <sarthak.0702@gmail.com>

mgr/dashboard: add snmp-gateway service e2e tests

Fixes: https://tracker.ceph.com/issues/54034
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #44934 from rhcs-dashboard/applitools-baseline-branch

mgr/dashboard: set appropriate baseline branch for applitools

Reviewed-by: aaryanporwal <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

rgw/dbstore: Add dbstore-tests to `make check`

Include and run dbstore-tests as part of `make check` target

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

mds: add an option to decide whether prefetching entire dir or not.
Accessing one single dentry could be fastened by set this option to
false, when dir is not in the memory.
Signed-off-by: "Shen, Hang" <shenhang@kuaishou.com>

mds: avoid adding null dentry to bloom filter

Signed-off-by: "Shen, Hang" <shenhang@kuaishou.com>

mds: make open_ino(...) fetch dentries in batch

Signed-off-by: "Shen, Hang" <shenhang@kuaishou.com>

mds: don't add null auth dentry to bottom lru

null dentries are the key of not fetching complete dirfrag. don't trim
them too fast

Signed-off-by: "Shen, Hang" <shenhang@kuaishou.com>

mds: fetch single dentry instead of complete dirfrag for lookup

Signed-off-by: "Shen, Hang" <shenhang@kuaishou.com>

mds: remove CDir::take_sub_waiting()

migrator already does the job when importing dirfrag.

Signed-off-by: "Shen, Hang" <shenhang@kuaishou.com>

mgr/dashboard: set appropriate baseline branch for applitools

All the dashboard PRs are checked against a baseline branch called
'default' in the visual regresstion testing. This will cause issues when
testing PRs in different branches. For eg: currently our master and
pacific has to save two different screenshots since the two of them
differ slightly.

Disabling the applitools logs as well because its too 'noisy'

Fixes: https://tracker.ceph.com/issues/54190
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #44477 from SMIL-Infra/fix-expire_recursive-segfault

mds: fix seg fault in expire_recursive

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44638 from yunfeiguan/wip-53911

mds: kill session state are open when mds do ms_handle_remote_reset

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #42549 from ajarr/wip-add-volume-rename

mgr/volumes: Add `fs volume rename` command

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44651 from nmshelke/fake-mount-53903

mount: add option to supports fake mounts in mount.ceph

Reviewed-by: Venky Shankar <vshankar@redhat.com>

rgw: Erase old storage class attr when the object is rewrited using radosgw-admin.

Fixes: https://tracker.ceph.com/issues/53790
Signed-off-by: zhiming zhang <zhangzhm1@chinatelecom.cn>

Merge pull request #44813 from soumyakoduri/wip-skoduri-dbstore-obj-id

rgw/dbstore: Use Object ID to handle I/O concurrency

Merge pull request #44678 from kipdoudou/master

rgw: CopyObject works with x-amz-copy-source-if-* headers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw/dbstore: Handle read vs delete races

Now that tail objects are associated with objectID, they are not deleted
as part of this DeleteObj operation. Such tail objects (with no head object
in *.object.table are cleaned up later by GC thread.

To avoid races between writes/reads & GC delete, mtime is maintained for each
tail object. This mtime is updated when tail object is written and also when
its corresponding head object is deleted (like here in this case).

Also done some cleanup.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

msg/async: fix outgoing_bl overflow and reset state_offset

    - we should reset state_offset when read done.

    - check outgoing_bl before we try to write a message.

    In some environments, network would temporily block and return EAGAIN.
    For async msgr, we would callback the write event directly, but that still
    increase the outgoing_bl.

    Think about this case, the sender is in congestion or network driver
    has some problems. The data appended to outgoing_bl and outgoing_bl
    is not consumed up-to-date immediately.
    That size of outgoing_bl will increase with time then overflow.
    The wrong outgoing_bl would cause some problems so we need to wait
    for outgoing_bl before we appended another message.

Signed-off-by: Vicente Cheng <vicente_cheng@bigtera.com>

Merge pull request #44930 from rishabh-d-dave/py-req-fix

Merge pull request #44667 from dvanders/dvanders_oft

mds: mds_oft_prefetch_dirfrags default to false

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44922 from ronen-fr/wip-rf-fix-bescrub-test

osd/scrub: fix unintended changes to scrub (cluster)logs

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #44927 from dang/wip-dang-cloud-tier

RGW - Move early return

Reviewed-by: Soumya Koduri <skoduri@redhat.com>

doc/man/8: Fix spelling in user rename example

Signed-off-by: Aggelos Toumasis toumasis.aggelos@gmail.com

osd/scrub: fix unintended changes to scrub (cluster)logs

OSD logs and cluster logs are monitored by some scrub tests:
some specific strings are required to either appear or not appear in
the logs. The Scrubber backend PR has unintentionally modified some
of these logs, and here we restore the exact logs text.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

rbd: mark optional positional arguments as such in help output

Currently at least five commands have optional positional arguments.

Overloading po::value<std::string>()->default_value("") for this
is a bit sneaky but nothing better fits into the existing Shell.cc
framework.

Note that strictly speaking "[<interval>] [<start-time>]" should be
"[<interval> [<start-time>]]" but we aren't doing that here because
"ceph" command doesn't do it either.

Fixes: https://tracker.ceph.com/issues/54191
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

monitoring: mention PyYAML only once in requirements

Following error occurs while running "sudo install-deps.sh" -
ERROR: Double requirement given: PyYAML==6.0 (from -r requirements-lint.txt (line 5)) (already in pyyaml (from -r requirements-alerts.txt (line 1)), name='PyYAML')

PyYAML is mentioned twice as a requirement. It is mentioned once in both
the following files -
monitoring/ceph-mixin/requirements-lint.txt
monitoring/ceph-mixin/requirements-alerts.txt

These requirements were added in commits
44d3e4c264506154373ffaeb13d6c924c580e6b5 and
4750ac0d7766a8a089adf073415af0ac0d3f81d9.

Fixes: https://tracker.ceph.com/issues/54185
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #44921 from cyx1231st/wip-seastore-cleanup-lba

crimson/os/seastore/lba_manager: consolidate logs

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44916 from joke-lee/dbstore-test-init-env

rgw: fix dbstore test init env wrong

crimson/os/seastore/lba_manager: introduce subsys seastore_lba_details

We will share the lba tree implementation, so classify the related logs
and rename to seastore_lba_details for now.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: consolidate logs

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #44713 from aclamk/fix-aclamk-vselector-53906

[BlueStore] Fix problem with volume selector

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #44261 from Matan-B/wip-matanb-c-msgr-network

test/crimson: Inject socket level random delays/failures msgr unit test

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #44856 from cbodley/wip-54103

qa/rgw: install s3cmd with extra_system_packages

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #44886 from holzman/bugfix-54128

krbd: return error when no initial monitor address found

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

krbd: return error when no initial monitor address found

Since we filter monitor addresses based on ms_mode, check that at
least one address was found.

Otherwise, we mismatch arguments when calling sysfs/add_single_major
which emits a misleading error message to dmesg:

  libceph: resolve 'name=user1' (ret=-3): failed
  libceph: parse_ips bad ip 'name=user1,key=client.user1'

Fixes: https://tracker.ceph.com/issues/54128
Signed-off-by: Burt Holzman <burt@fnal.gov>

RGW - Move early return

When transitioning an object to cloud, there was an early return,
skipping the removal of the cloud target.  Fix this to be in the right
place.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #44825 from rhcs-dashboard/rgw-dashboard-test-hanging

mgr/dashboard: fix for cephadm e2e failing because of rgw commands getting stuck

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44920 from rhcs-dashboard/cephadm-bootstrap-fix

cephadm: change shared_folder directory for prometheus and grafana

Reviewed-by: MrFreezeex <NOT@FOUND>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

crimson/os/seastore/lba_manager: convert to seastore logging

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

cephadm: change shared_folder directory for prometheus and grafana

After https://github.com/ceph/ceph/pull/44059 the monitoring/prometheus
and monitoring/grafana/dashboards directories are changed to
monitoring/ceph-mixins. That broke the shared_folders in the cephadm
bootstrap script.

Changed all the instances of monitoring/prometheus and
monitoring/grafana/dashboards to monitoring/ceph-mixins

Also, renaming all the instances of prometheus_alerts.yaml to
prometheus_alerts.yml.

Fixes: https://tracker.ceph.com/issues/54176
Signed-off-by: Nizamudeen A <nia@redhat.com>

crimson/os/seastore/lba_manager: move lba_tree_inner_stats into LBAManager

Cleanup, change stats from LBATree static member to BtreeLBAManager
class member.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

rgw: fix dbstore test init env wrong

fix https://tracker.ceph.com/issues/54174

Signed-off-by: yuliyang_yewu <yuliyang_yewu@cmss.chinamobile.com>

Merge pull request #44503 from lxbsz/typo1

mds: fix a typo of inode version related comment

Reviewed-by: Venky Shankar <vshankar@redhat.com>

rgw/dbstore: Create GC Thread to delete stale objects

Create a gc thread to cleanup the stale tail objects data

XXX: handle read + delete usecase, simple approach could be
to use locks or sqlite transactions in GC

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/dbstore: Use Object ID to handle racing writes

Create unique ID for each object upload which will be
atomically updated in the head object at the end. This will
prevent data corruption during concurrent writes.
Incase of Multipart Uploads, upload_id is used as ObjectID.

XXX: The stale or obsolete tail data needs to be deleted

Also addressed invalid usage of CephContext in dbstore tests.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

test/msgr: add unittest to simulate network block temporarily

   Add new test case to verify the network block temporarily,
   that case would make outgoing_bl overflow so add the assert
   checking mechanism to claim_append

   Just use 2 connections because that we could not generate the
   large data set to verify it

   Simulate the EAGAIN situation looks like by skip calling
   cs.send() because EAGAIN would return size 0 and keep the
   outgoing_bl

Signed-off-by: Vicente Cheng <vicente_cheng@bigtera.com>

Merge pull request #44842 from idryomov/wip-krbd-rxbounce-option

rbd: recognize rxbounce map option

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #43910 from sunnyku/wip-rbd-snap-sched

mgr/rbd_support: fix schedule remove

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>