osd, mon: Add 'osd pg-upmap-primary' and 'osd rm-pg-upmap-primary' commands.

Changed OSDMap to keep upmap-primary records
The upmap-primary records are visible in json/xml and in dump commands.

Signed-off-by: Josh Salomon <jsalomon@redhat.com>

Merge pull request #48699 from zdover23/wip-doc-2022-11-02-dev-osd-internals-ec-glossary-fixes

doc/dev: refine erasure_coding.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev: refine erasure_coding.rst

Improve the readability and clarity of
erasure_coding.rst.

Co-author: Cole Mitchell <cole.mitchell@gmail.com>
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48687 from StevenWash/bugfix/fix_command_error_blank

doc/radosgw: fix command error blank

doc/radosgw: fix command error blank

Merge pull request #48684 from zdover23/wip-doc-2022-11-01-dev-osd-internals-ec-glossary-alphabetize

doc/dev: alphabetize EC glossary

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/dev: alphabetize EC glossary

Alphabetize the glossary in
doc/dev/osd_internals/erasure_coding.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48673 from idryomov/wip-49947

doc: note architecture availability for PMEM mode of PWL cache

Reviewed-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #48548 from dparmar18/update_debug_script

script/ceph-debug-docker.sh: change default branch to main

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #48674 from zdover23/doc-2022-10-31-dev-osd-internals-ec-glossary

doc/dev: improve EC glossary

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/dev: improve EC glossary

Improve the clarity and syntax of the text in
doc/dev/osd_internals/erasure_coding.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc: note architecture availability for PMEM mode of PWL cache

Fixes: https://tracker.ceph.com/issues/49947
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #48667 from zdover23/wip-doc-2022-10-30-ceph-volume-lvm-prepare-colemitchell-notes

doc/ceph-volume: refine Filestore docs

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/ceph-volume: refine Filestore docs

Add Cole Mitchell's improvements to the filestore section.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48665 from zdover23/wip-doc-2022-10-30-ceph-volume-lvm-prepare-datri-notes

doc/ceph-volume: improve prepare.rst

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/ceph-volume: improve prepare.rst

Incorporate Anthony D'Atri's suggestions from
https://github.com/ceph/ceph/pull/48651.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48090 from joscollin/wip-F55197-cephfs-top-scroll

cephfs-top: make cephfs-top display scrollable

Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #48662 from ceph/wip-update-stale-bot

.github/workflows: update `days-before-pr-close`

.github/workflows: update `days-before-pr-close`

Currently, the stale bot closes PRs 90 days after they are marked stale since it counts "marking a PR stale" as activity.

We really want the PRs to close after a total of 90 days, or 30 days after they are marked stale. Changing `days-before-prs-close` to 30 will fix this.

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #48658 from zdover23/wip-doc-2022-10-28-radosgw-multisite-prompts

doc/radosgw: add prompts to multisite.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: add prompts to multisite.rst

Add unselectable prompts to multisite.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #47582 from snosratiershad/docs/correct-add-system-user-to-the-master-zone-command

docs: correct add system user to the master zone command

Reviewed-by: Zac Dover <zac.dover@gmail.com>

cephfs-top: Select Filesystem Screen fixes for no FS

* drop the blank FS Menu when there is no FS
* cleanup display_fs_menu() code

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc: Update cephfs-top.png

Fixes: https://tracker.ceph.com/issues/55197
Signed-off-by: Jos Collin <jcollin@redhat.com>

cephfs-top: make cephfs-top display scrollable

Fixes: https://tracker.ceph.com/issues/55197
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #40363 from orozery/rbd-clone-encryption

librbd: add encryption format support for clones (part 1/2)

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

script/ceph-debug-docker.sh: change default branch to main

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #48347 from ronen-fr/wip-rf-acting

osd/scrub: use the actual active set when requesting replicas to scrub

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #48510 from ceph/update-mailmap

mailmap: add Pedro Gonzalez Gomez

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #48528 from ceph/fix-gh_pr_triage_node_version-main

.github/pr-triage: update node.js to v16

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #48515 from athanatos/sjust/wip-crimson-suites

qa/suites/crimson-rados: move seastore to crimson-rados-experimental, remove cache/dedup workloads

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #48533 from zdover23/wip-doc-2022-10-19-radosgw-notification-topic-management-and-notification-performance

doc/radosgw: refine top. man. and not. perf. stats

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: refine top. man. and not. perf. stats

This commit refines the English in the "Topic Management via CLI" and
"Notification Performance Statistics" sections of the notifications.rst
file.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48444 from yuvalif/wip-yuval-fix-57850

rgw/amqp: store CA location string in connection object

qa/suites/crimson-rados: remove cache and dedup related tests

It's an accident that some of these pass -- the tests provide some
coverage for inconsistencies that can arise, but don't really validate
that the objects actually get moved between tiers.

It's going to be some time before we implement cache or dedup in
crimson, and we'll probably want to disable the related commands for
pools that can map to crimson osds to prevent accidents.

Signed-off-by: Samuel Just <sjust@redhat.com>

qa/suites: move seastore to crimson-rados-experimental suite

Let's use crimson-rados-experimental for tests that don't yet
pass reliably.  We can move these to crimson-rados as they
become reliable.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #48511 from zdover23/wip-doc-2022-10-18-radosgw-notification-reliability

doc/radosgw: refine "notification reliability"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: refine "notification reliability"

This commit refines the English in the "Notification Reliability"
section of the notifications.rst page (the "Bucket Notifications"
page). I have also added signposting structure.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Co-authored-by: Yuval Lifshitz <ylifshit@redhat.com>
Signed-off-by: Zac Dover <zac.dover@gmail.com>

.github/pr-triage: update node.js to v16

[Warning](https://github.com/ceph/ceph/actions/runs/3273201887):
```
Node.js 12 actions are deprecated. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/.
Please update the following actions to use Node.js 16: actions/labeler, iyu/actions-milestone
```

Signed-off-by: Ernesto Puerta <37327689+epuertat@users.noreply.github.com>

Merge pull request #48484 from cyx1231st/wip-crimson-msgr-cleanup

crimson/net: simplify msgr interfaces with AuthClient/Server

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48514 from ceph/wip-dashboard_gh_project-main

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #48485 from vshankar/wip-upgrade-x-o-q

qa: fs:upgrade featureful_client test upgrades to quincy

Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #48398 from rhcs-dashboard/fix-get-rbd-stats

mgr/prometheus: avoid duplicates and deleted entries for rbd_stats_pools

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #48471 from SUSE/wip-fix-57860

ceph.spec.in: disable system_pmdk on s390x for SUSE distros

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

.github/pr-triage: switch from old to new project

The old (classic) Dashboard project (https://github.com/ceph/ceph/projects/6) has been migrated to a new-style one (https://github.com/orgs/ceph/projects/2/views/1).

This also requires updating the GH Action that automatically assigned `dashboard`-labeled PRs to the Dashboard project.

Signed-off-by: Ernesto Puerta <37327689+epuertat@users.noreply.github.com>

Merge pull request #48512 from zdover23/wip-doc-2022-10-18-radosgw-notifications-prompts

doc/radowsgw: add prompts to notifications.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

mailmap: add Pedro Gonzalez Gomez

Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

doc/radowsgw: add prompts to notifications.rst

This PR adds unselectable prompts to doc/radosgw/notifications.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48241 from ajarr/fix-57210

 src/mds: increment directory inode's change attr by one

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #48425 from phlogistonjohn/jjm-shell-fixups

build: minor build script cleanups

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48501 from zdover23/wip-doc-2022-10-17-radosgw-notifications

doc/radosgw: refine notifications.rst - top part

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/radosgw: refine notifications.rst - top part

This commit refines the English in the first part of the
notifications.rst page (the "Bucket Notifications" page).

Signed-off-by: Zac Dover <zac.dover@gmail.com>

install-deps: switch to C.UTF-8 locale

The en_US.UTF-8 locale is not always going to present on all systems.
Specifically, setting en_US.UTF-8 causes numerous warnings when running
this script on a CentOS Stream 8 container. Rather than install
additional packages to suppress the warning, use the C.UTF-8 locale.

Additionally, try to update the comment explaining the line to something
based on the commit history rather than the confusing term "vulnerable".

Signed-off-by: John Mulligan <jmulligan@redhat.com>

script/run-make: convert a tab to spaces

There was a single line using a hard tab to indent, convert the tab
and reformat the spacing of the line near it.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

install-deps: for consistency, reformat tabs to spaces

The majority of the install-deps.sh file was already using spaces to
indent, convert the remaining hard tabs to 8 spaces.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

install-deps: remove emacs modeline from script

The majority of the lines in this shell script uses spaces to indent,
but I *think* this modeline tells emacs to indent with tabs. In a very
near future change I'm going to make all indents use spaces. So we first
remove the emacs modeline that tells it to use the opposite.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #48086 from vshankar/wip-57084

client: use parent directory POSIX ACLs for snapshot dir

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #48497 from liu-chunmei/fix-variable-usage-after-move

crimson/osd: not use a moved variable

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #48498 from zdover23/wip-doc-2022-10-16-radosgw-notifications-wording

doc/radosgw: update notifications.rst - grammar

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/radosgw: update notifications.rst - grammar

This commit rewrites a sentence, as requested by
C. Mitchell here:
https://github.com/ceph/ceph/pull/48493/files#r996003812

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson/osd: not use a moved variable

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

cephfs: add additional logging to track change_attr of inodes

Signed-off-by: Ramana Raja <rraja@redhat.com>

src/mds: increment directory inode's change attr by one

... whenever the mtime or ctime of the directory inode is modified.

In CephFS subvolume clones exported using NFS-Ganesha, newly created
files using `touch` were not being listed. It was identified that the
create request sent to the Ceph MDS via NFS-Ganesha's libcephfs client
modified the mtime and ctime of the parent directory, but did not modify
the change_attr of the parent directory. Since the NFS client
didn't see a modification of the change attribute in the reply, it
didn't invalidate its readdir cache. The subsequent directory `ls` was
satisfied from the NFS client's stale readdir cache.

Whenever parent directory inode's  mtime was modified in
MDCache::predirty_journal_parents(), the parent inode's change_attr
was set to its dirstat->change_attr. The parent inode's
dirstat->change_attr doesn't track changes to parent's *ctime only*
changes such as setattr, setvxattr, etc. on the parent
directory. See commit 0d441dcd6af553d11d6be6df56d577c5659904a0 for more
details. This caused the directory inode's change_attr to not be updated
when an operation to change only its ctime was followed by an operation
to change its mtime and ctime.

Fix this by making changes to MDCache::predirty_journal_parents() and
CInode::finish_scatter_gather_update() to increment the directory
inode's change_attr by one instead of setting it to its
dirstat->change_attr.

Fixes: https://tracker.ceph.com/issues/57210
Signed-off-by: Ramana Raja <rraja@redhat.com>

test/libcephfs: enhance directory's change_attr testing

Alternate operations that only change directory's ctime
(setattr/setxattr/removexattr on directory) with those that change
directory's mtime and ctime (create/rename/remove a file within
directory). Check that directory's change_attr is updated everytime
ctime changes.

Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #48227 from phlogistonjohn/jjm-cephadm-test-nfsg

cephadm: add test coverage for NFSGanesha, HAproxy, & Keepalived classes

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #48493 from zdover23/wip-doc-2022-10-15-radosgw-notifications-cleanup

doc/radosgw: improve grammar - notifications.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: improve grammar - notifications.rst

This commit makes the changes suggested by C. Mitchell in
https://github.com/ceph/ceph/pull/48486.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48422 from aravind-wdc/refactor

crimson/zns: refactor zone operations

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48486 from zdover23/wip-doc-2022-10-14-radosgw-push_endpoint

doc/radosgw: add push_endpoint for rabbitmq

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

crimson/zns: refactor zone operation

Moved all zone operations into one routine.

Signed-off-by: Aravind Ramesh <Aravind.Ramesh@wdc.com>

doc/radosgw: add push_endpoint for rabbitmq

This commit directs users of rabbitmq to use "push_endpoint"
(with an underscore) instead of "push-endpoint" (with a hy-
phen). This commit adds a note that contains such a direct-
ive. It does not alter the examples already present in the
text.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

qa: fs:upgrade featureful_client test upgrades to quincy

Reef cycle will not allow upgrades from octopus. However, the featureful
client tests still needs to be testes, therefore, upgrade to quincy (from
octopus) rather to the current cycle (reef).

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #48475 from rzarzynski/wip-crimson-resolve_oid-clone_snaps-fix-search

crimson/osd: resolve_oid() looks in clone_snaps by requested snap ID

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #48405 from athanatos/sjust/wip-57759

crimson: tmap support, list_snaps support, fixes for zero and cmpext

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

mgr/prometheus: avoid duplicates and deleted entries for rbd_stats_pool

Fixes: https://tracker.ceph.com/issues/57797
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Avoid duplicate entries (pool[/namespace]) for rbd_stats_pools config and also avoid deleted pools.

crimson/osd: resolve_oid() looks in clone_snaps by requested snap ID

Before the patch the method was looking for `lower_bound(oid.snap)`
from `SnapSet::clones` which leads to reading from head instead of
returning the `ENOENT` expected in `TestLibRBD.TestIOToSnapshot`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: hide peer_global_id from Connection class

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: replace unnecessary ConnectionRef/AuthConnectionMetaRef

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: drop Connection::get_messenger()

Simplify Connection interface for multi-core support.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: drop require_authorizer feature

There is no need to consider the pre-nautilus scenario.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: drop the unused ProtocolV1

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

ceph.spec.in: disable system_pmdk on s390x for SUSE distros

Fixes: https://tracker.ceph.com/issues/57860
Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #48437 from rosinL/fix-vstart

src/vstart.sh: correct top_cpu's value

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48457 from cyx1231st/wip-fix-msgr-test

test/crimson/test_messenger: add pause before marking down a connection

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #48463 from zdover23/wip-doc-2022-10-14-dev-dev-guide-basic-workflow-submodule-update-2

doc/dev: add submodule-update link to dev guide

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/dev: add submodule-update link to dev guide

This commit links to the procedure in install/clone-source.rst that
explains how to update submodules.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48460 from zdover23/wip-doc-2022-10-14-install-clone-source-submodules

doc/install: improve updating submodules procedure

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

crimson/osd: CEPH_OSD_OP_ZERO should become truncate if past end of object

librbd image diff seems to rely on this behavior to modify oi.size for
LIST_SNAPS.

Fixes: https://tracker.ceph.com/issues/57791
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: add CEPH_OSD_OP_LIST_SNAPS

Fixes: https://tracker.ceph.com/issues/57789
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #47853 from WOnder93/selinux-io_uring

selinux: prepare for anon inode controls enablement

doc/install: improve updating submodules procedure

This commit beefs up the "Updating Submodules" procedure,
which (prior to this commit) was quite brief.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson/net: print connection memory address as the identity

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson/test_messenger: add pause before marking down a connection

To make sure the ACK messages are consumed, otherwise there can be
unexpected connect attempts to break the test.

Fixes: https://tracker.ceph.com/issues/57693
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #48377 from ktdreyer/win-deps-clone

win32: speed up and simplify deps cloning

Merge pull request #48406 from athanatos/sjust/wip-57798

qa/suites/crimson-rados: add CRIMSON_COMPAT to workunit env

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #48446 from zdover23/wip-doc-2022-10-12-glossary-ceph-monitor

doc/glossary: add "ceph monitor" entry

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: add "ceph monitor" entry

This commit adds information to the "ceph monitor" entry in
glossary.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

rgw/amqp: store CA location string in connection object

Fixes: https://tracker.ceph.com/issues/57850
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #48196 from neesingh-rh/wip-human-readable-volume-info

mgr/volumes: Add human-readable flag to volume info command

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #48367 from stefan-chivu/namespace-optarg-fix

rbd: device map/unmap --namespace handling fixes

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>