install-deps.sh: use ceph-libboost1.73

Fixes: https://tracker.ceph.com/issues/46208
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #31726 into master

* refs/pull/31726/head:
mds: MDS-MDS message classes inherit from MMDSOp
mds: MMDSOp implementation and detect unversioned messages

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #34878 into master

* refs/pull/34878/head:
libcephfs: mark ceph_stat() family functions deprecated

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #35323 into master

* refs/pull/35323/head:
tool: change coding style, indent with two spaces

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #35420 into master

* refs/pull/35420/head:
mgr/volumes: Fix pool removal on volume deletion

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #35624 into master

* refs/pull/35624/head:
include: cleanup lru con/des

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #35664 into master

* refs/pull/35664/head:
qa: add omit_sudo=False for commands ran with sudo

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #35757 into master

* refs/pull/35757/head:
doc: update cephfs metadata pool recovery procedure

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #35537 from sebastian-philipp/cephadm-yaml-ordered-readable

mgr/orch: increase readability for yaml representation

Merge pull request #35274 from jmolmo/issue_45726

cephadm: error trying to get ceph auth entry for crash daemon

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35773 from theanalyst/master-rgw-cors-fixes

rgw: sanitize newlines in s3 CORSConfiguration's ExposeHeader

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Josh Durgin <jdurgin@redhat.com>

doc: update cephfs metadata pool recovery procedure

This is mostly clarifying a few things after personally experimenting
with this.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35695 from sebastian-philipp/mypy-rm-global-ignore-imports

mypy.ini: Remove global `ignore_missing_imports = True`

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #35762 from sebastian-philipp/doc-dev-rm-confusing.txt

doc/dev: remove confusing.txt

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35503 from sebastian-philipp/rm-daemon-ok-to-stop

mgr/cephadm: move ok_to_stop to CephadmService

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35761 from sebastian-philipp/qa-cephad-upgrade-no-rgw

qa/cephadm/upgrade: Exclude RGW for now

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33228 from galsalomon66/s3select_integration_into_RGW_2

rgw: add s3select support

Merge pull request #35593 from p-se/wip-pse-cephadm-grafana-fix-version

mgr/cephadm: use fixed Grafana version

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34606 from rhcs-dashboard/wip-45011-bucket-usage-feature

mgr/dashboard: Display users current bucket quota usage

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

rgw: sanitize newlines in s3 CORSConfiguration's ExposeHeader

the values in the <ExposeHeader> element are sent back to clients in a
Access-Control-Expose-Headers response header. if the values are allowed
to have newlines in them, they can be used to inject arbitrary response
headers

this issue only affects s3, which gets these values from an xml document

in swift, they're given in the request header
X-Container-Meta-Access-Control-Expose-Headers, so the value itself
cannot contain newlines

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Reported-by: Adam Mohammed <amohammed@linode.com>

Merge pull request #35419 from votdev/issue_45897_add_host_labels

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35767 from tchaikov/wip-rpm-scikit-learn

ceph.spec.in: Requires python3-scikit-learn only on fedora and suse

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

python-common: service_spec.from_json: Add docstring explaining histroy

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/cephadm: Add test for --format=yaml

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Add test for readable yaml

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr: move test_orchestrator to orchestrator module

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: make YAML of ServiceDescription readable

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: YAML of DaemonDescription readable

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common: Make YAML or OSD Specs readable

* Changes: An empty OSD Spec is now invalid.
* OSDSpec.validate() now fails, if service-id is empty

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common: make YAML representaition of ServiceSpec readable

* Add test for new yaml representation

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

ceph.spec.in: Requires python3-scikit-learn only on fedora and suse

so far, python3-scikit-learn is not packaged by RHEL8/CentOS8 or EPEL8.

see
https://src.fedoraproject.org/rpms/python-scikit-learn/branches?branchname=master

this change should address the regression introduced by
d13b441cdb63e37ad3b8ba36cbd3edc33b1e4bbd

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/cephadm/upgrade: use quay.ceph.io

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/cephadm/upgrade: Exclude RGW for now

tasks/cephadm.py gained RGW support very recently and
I'm now facing a dilemman:

* Either we set the upgrade start to 15.2.4 and thus
  no longer upgrade from an old version, or
* Disable RGW upgrade for now.

I think doing both would be optinal, but for now, let's
disable RGW, in order to keep the coverage for everything
else.

Fixes: https://tracker.ceph.com/issues/46157
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35292 from s0nea/wip-dashboard-telemetry-notification

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

doc/dev: remove confusing.txt

* Last change was from 2011
* the file is not referenced anywhere
* the referenced file `cephconfig.jpg` doesn't exist

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35648 from ideepika/fix-44948

ceph.spec: fix mgr diskprediction_local dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35722 from ideepika/drop-el7

ceph.spec: drop support for el7 rpm builds

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35534 from varshar16/wip-doc-fix-SyntaxWarning

doc/scripts/gen_state_diagram.py: Fix literal comparison syntax warnings

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35677 from sebastian-philipp/_refresh_hosts_and_daemons_forall_hosts

mgr/cephadm: speedup _refresh_hosts_and_daemons by 10x

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/dashboard: Display users current bucket quota usage
Fixes: https://tracker.ceph.com/issues/45011
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit 966d887f7a5e0765b4210ab4edcb1cea3e03ac35)

Merge PR #30592 into master

* refs/pull/30592/head:
qa: fix flake8 warnings
doc: add documentation for new ephemeral pinning feature
mds: enable ephemeral pinning by default
pybind/mgr/volumes: wire up pinning subvolumes/subvolumegroups
qa: adapt tests for empty pinned dir export
qa: break export pin tests into discrete tests
qa: add more ephemeral pin tests
qa: add tests for ephemeral pinning
mds: add maximum random ephemeral pin percentage
mds: replicate random pin state
mds: finish implementation of ephemeral pins
mds: do string equality comparison
mds: add ephemeral pinning for subtrees
mds: trim pinned and empty subtrees
mds: refactor remove_subtree
mds: allow export of pinned directory if empty
mds: reduce subtree processing verbosity
mds: skip export of empty directories
mds: remove frozen export pin from queue
mds: simplify for loop construction
mds: add debug messages for export queue processing
qa: refactor _wait_subtree and _get_subtree
qa: use status from wait_for_daemons
qa: quietly print json output from asok commands

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35736 from ivancich/wip-orphan-list-timestamp-fix

rgw: orphan-list timestamp fix

Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

qa: fix flake8 warnings

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: add documentation for new ephemeral pinning feature

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: enable ephemeral pinning by default

This feature is enabled by default for Pacific. This commit will be
dropped in the backport to Octopus.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/volumes: wire up pinning subvolumes/subvolumegroups

Fixes: https://tracker.ceph.com/issues/41541
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: adapt tests for empty pinned dir export

Previously, empty pinned directories were not migrated.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: break export pin tests into discrete tests

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add more ephemeral pin tests

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add tests for ephemeral pinning

Signed-off-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

mds: add maximum random ephemeral pin percentage

This new config is designed to prevent creating too many subtrees from a
random ephemeral pinning policy.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: replicate random pin state

This is slightly evil in its current form. The MDS should use locks to
transmit state changes but right now it's just set when the CInode is
replicated. This replication of this state marker is necessary for
failover situations where we want the randomly pinned subtree to remain
pinned across failovers.

Note: this problem does not exist for the ephemeral distributed pins
because simple knowledge of the immediate parent's setting (which is
replicated normally) is sufficient to determine if the CInode is
ephemerally distributed. Ditto for regular export pins.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: finish implementation of ephemeral pins

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: do string equality comparison

The string::find method would return true for ceph.dir.pin even for the
other ephemeral pin xattr names. For this reason, it was never possible
to actually turn ephemeral pins on!

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add ephemeral pinning for subtrees

This PR introduces inode xattrs export_ephemeral_random and
export_ephemeral_distributed which enables two different metadata
distribution strategies - the first being suitable for a more depthwise
scaling of metadata (height of the tree keeps increasing) and the latter
for horizontal scaling (many subtrees under a single parent).
export_ephemeral_distributed applies is not hierarchical. Any direct
descendant directory (i.e. a child directory) has an ephemeral export
pin applied to it according to a consistent hash of the child directory
inode number. export_ephemeral_distributed is hierarchical like
"export_pin". Any CDir loaded into the cache may be ephemerally pinned
to a random rank. Like "export_ephemeral_distributed", the random rank
is determined by a consistent hash.

The metadata distribution strategies are facilitated by using John
Lamping and Eric Veach's Jump Consistent Hashing as the consistent hash
algorithm. This hashing algorithm eliminates the need to store the data
structures representing the consistent hash cluster state and performs
as well as Akamai's original implementation providing a fairly uniform
distribution. This algorithm only works for distributed systems with
numbered buckets (nodes) arranged in ascending order and cluster resizes
does not produce any holes in the arrangement of nodes i.e (0, 1, 2, 3)
--[removing node 1]--> (0, 1, 2). CephFS satisfies these conditions as
the MDSs are arranged as numbered ranks and cluster modifications does
not produce any holes in the resulting arrangement of ranks.

Fixes: https://tracker.ceph.com/issues/41302
Signed-off-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: trim pinned and empty subtrees

Before export (and ephemeral) pinned subtrees are stuck in cache
forever.

Add qa test for checking export pinned directories can be trimmed.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: refactor remove_subtree

Search each map only once as necessary.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: allow export of pinned directory if empty

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: reduce subtree processing verbosity

and some mild loop refactoring.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: skip export of empty directories

Note: empty as in no cached sub-entries.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: remove frozen export pin from queue

The export already belongs on the node id so there's no need to keep it
in the queue.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: simplify for loop construction

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add debug messages for export queue processing

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: refactor _wait_subtree and _get_subtree

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use status from wait_for_daemons

Avoid an extra `fs dump` call.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: quietly print json output from asok commands

Pretty print output once. Use --format=json so the stdout on teuthology
is not pretty printed, taking hundreds of lines.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35752 into master

* refs/pull/35752/head:
qa: fix flake8 error

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa: fix flake8 error

    ./tasks/vstart_runner.py:33:1: F401 'io.BytesIO' imported but unused

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35665 into master

* refs/pull/35665/head:
vstart_runner.py: improve a conditional in _do_run

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33879 into master

* refs/pull/33879/head:
client: fix snap directory atime

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35518 into master

* refs/pull/35518/head:
mds: fix filelock state when Fsc is issued

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #35522 into master

* refs/pull/35522/head:
vstart_runner: set default values of stdout and stderr to None

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35532 into master

* refs/pull/35532/head:
Revert "Revert "qa/suites/rados/mgr/tasks/module_selftest: whitelist mgr client getting backlisted""

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #35540 into master

* refs/pull/35540/head:
qa/cephfs: don't pass cmd args from run_as_user as str
qa/cephfs: refactor run_as_root() to user run_as_user()

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35582 into master

* refs/pull/35582/head:
mds: reset heartbeat in EMetaBlob replay

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35619 into master

* refs/pull/35619/head:
mds: reset sequence number when new rank0 mds is available
mds: dump mds name when processing mds map

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: orphan-list timestamp fix

When creating intermediate and output files, the rgw-orphan-list
script uses a timestamp using the `date` command. The hour was
inserted with "%k" but that padds with a space rather than a zero. So
that's changed to "%H".

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #35208 from prateek2211/refactor

rgw: Replace boost::string_ref/view with std::string_view

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35331 from yuvalif/fix_issue_43536

rgw/pubsub: remove deprecated radogw-admin pubsub creation commands

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #35604 into master

* refs/pull/35604/head:
CODEOWNERS: added doc-writers team

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #35739 into master

* refs/pull/35739/head:
mgr/volumes/nfs: Reorder imports

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35716 from rzarzynski/wip-bl-rval-claim_append

common, msgr/async: bufferlist::claim_append() doesn't require intermediaries anymore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35643 from sebastian-philipp/cephadm-fix-already-in-use-msg

mgr/cephadm: fix broken "already in use" exception message

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35633 from matthewoliver/cephadm_bootstap_ipv6

cephadm: Set ms bind ipv6 when mon-ip is ipv6

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35589 from jschmid1/osd_servicedescription_fix

mgr/cephadm: exclude osds without osdspec_affinity

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35655 from batrick/i46081

cephadm: restrict mds caps to cephfs pools

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35651 from sebastian-philipp/cephadm-unit-run-set-e

cephadm: unit.run: add `set -e`

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #35654 from neha-ojha/wip-44186

pybind/mgr/pg_autoscaler/module.py: do not update event if ev.pg_num== ev.pg_num_target

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35710 from rhcs-dashboard/wip-46144-fix-sort-ui

mgr/dashboard: fix UI for Device class column in OSDs list

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35742 from theanalyst/doc/releases/oct-update

doc: releases: updates for octopus

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #35721 from bk201/wip-43971

mgr/dashboard: fix Source column i18n issue in RBD configuration tables

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #35734 from tspmelo/wip-fix-tooltip

mgr/dashboard: Fix display of tooltips inside tables

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

doc: releases: octopus: fixup changelog

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: releases: update timeline for octopus

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/volumes/nfs: Reorder imports

This patch reorders import to fix "ImportError: cannot import name 'TimedOut'".
As flake8 requires similar import types to be grouped together.

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/dashboard: Bypass current selection when calling CdTableAction::disableDesc().

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Add host labels in UI

Fixes: https://tracker.ceph.com/issues/45897
Signed-off-by: Volker Theile <vtheile@suse.com>
s

s

s

CODEOWNERS: added doc-writers team

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #35473 from aclamk/fix-45903-bluefs-log-growth

os/bluestore: fix bluefs log growth.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

doc: update releases table

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>