mgr/cephadm: Fix test_facts

Wasn't executed before

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44293 from sebastian-philipp/extract-uid-gid-readable

cephadm: make extract_uid_gid errors more readable

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #44208 into master

* refs/pull/44208/head:
mgr/progress: avoid inefficient dump of all pg stats

Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #44211 into master

* refs/pull/44211/head:
mon: increase mon_down_mkfs_grace to 2m

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #43889 from Daniel-Pivonka/cephadm-reg-cred

mgr/cephadm: store contianer registry credentials in config-key

Reviewed-by: Sage Weil <sage@newdream.net>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44031 from adk3798/agent-thrash

 mgr/cephadm: agent: simplify handling of agent reports

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44267 from phlogistonjohn/jjm-ceph-orch-yaml-issue206961

python-common: add int value validation for count and count_per_host

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #44269 from SUSE/wip-fix-mgr-cephadm-suse-cherrypy-requires

ceph.spec.in: fix mgr-cephadm CherryPy requirement for SUSE builds

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #44197 from josephsawaya/zns-segment-manager

crimson/os/seastore: add ZNSSegmentManager

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44323 from cbodley/wip-53252

rgw/amqp: add default case to silence compiler warning

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

mgr/cephadm: agent: simplify handling of agent reports

Don't try to do extra things like checking other agents
or updating health checks when agents report. Rely on
serve loop for that

This also should help with thrashing we've been seeing

Signed-off-by: Adam King <adking@redhat.com>

rgw/amqp: add default case to silence compiler warning

Fixes: https://tracker.ceph.com/issues/53252
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44131 from ceph/wip-yuriw-release-16.2.7-master

doc: 16.2.7 Release Notes

Merge pull request #44136 from adk3798/agent-logging

mgr/cephadm: agent: log response from mgr

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44249 from DimitriPapadopoulos/codespell

doc,man: Typos found by codespell

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44309 from foadlind/fix_cephadm_doc

doc/cephadm/upgrade: correct example command

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc,man: typos found by codespell

Signed-off-by: Dimitri Papadopoulos <3234522+DimitriPapadopoulos@users.noreply.github.com>

Merge pull request #44216 from locallocal/master

os/bluestore: don't need separate variable to mark hits when lookup oid.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #43770 from ifed01/wip-ifed-fix-53002

os/bluestore: avoid premature onode release.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #43412 from yaarith/devicehealth_fix_tz

mgr/devicehealth: fix missing timezone from time delta calculation

Reviewed-by: Laura Flores <lflores@redhat.com>

crimson/os/seastore: add ZNSSegmentManager

This commit adds the ZNSSegmentManager, so SeaStore can
interface with ZNS devices using the linux/blkzoned
library.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

doc/releases/pacific.rst: Instructions for notable change

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #44163 from rhcs-dashboard/fix-53451-master

mgr/dashboard: disable Promql test in ARM

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: p-se <NOT@FOUND>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>

Merge pull request #44295 from chrisphoffman/add-email-contact

mailmap: Add Christopher Hoffman

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44299 from kalebskeithley/dencode2

rgw:cleanup/refactor json and xml encoders and decoders

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

os/bluestore: avoid premature onode release.

This was observed when onode's removal is followed by reading
and the latter causes object release before the removal is finalized.
The root cause is an improper 'pinned' state assessment in Onode::get

More detailed overview is:
At some point Onode::get() might face the case when nref == 2 and pinned = true
which means parallel incomplete put is running on the onode - ref count is
decremented but pinned state is still unmodified (and even lock hasn't been
acquired yet).
This might finally result in two puts racing over the same onode with nref == 2
which finally results in a premature onode release:
  // nref =3, pinned = 1
  // Thread 1                   Thread 2
  //   o->put()                   o->get()
  //   --nref(n = 2, pinned=1)
  //                              nref++ (n=3, pinned = 1)
  //                              return
  //                              ...
  //                              o->put()
  //                              --nref(n = 2)
  //                              pinned = 0,
  //                              --nref(n = 1)
  //                              ocs->_unpin_and_rm(o) -> o->put()
  //                                ...
  //                                --nref(n = 0)
  //                                release o
  //  o->c->get_onode_cache()
  //  FAULT!
  //
The suggested fix is to introduce additional atomic counter tracking
running put() functions. And permit onode release when both regular
nref and put_nref are both equal to zero.

Fixes: https://tracker.ceph.com/issues/53002
Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #44029 from cbodley/wip-rgw-beast-header-limit

rgw/beast: add max_header_size option with 16k default, up from 4k

Reviewed-by: Mark Kogan <mkogan@redhat.com>

cephadm: make extract_uid_gid errors more readable

Avoid dumping a traceback

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm/upgrade: correct example command

Update the ceph version used in the example upgrade command to match the one mentioned in the text above it.

Signed-off-by: Foad Lind <foad.lind@citynetwork.eu>

Merge pull request #44235 from xxhdx1985126/wip-onode-omap-hint-optimization

crimson/os/seastore: avoid onode/omap laddr hint conflicts as much as possible

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

crimson/os/seastore: make onode data/metadata laddr space reservation configurable

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: randomize metadata laddr hints

This should prevent omap and xattr extent allocations from clumping near
the onode's hint.  Additionally, only generate them past the default
16MB object_data_handler reservation.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common: DONOT call crimson::get_logger() if NDEBUG is defined

Avoid debug related perf degredation

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #44141 from xxhdx1985126/wip-53409

crimson/os/seastore/segment_cleaner: correct available space calculation

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #44290 from liu-chunmei/crimson-fix-no-pg

crimson/osd: fix interruptor assert when no pg in peering_event

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: correct available space calculation

Current available space calculation is wrong, it just counts the space occupied
by extents, deltas and other stuff are not taken into account.

Fixes: https://tracker.ceph.com/issues/53409
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/common: redirect interruptible future debug output to default subsys

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/segment_cleaner: add perf metrics for better monitoring

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #44015 from liewegas/fix-44012

osd/PeeringState: separate history's pruub from pg's

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43864 from yaarith/fix-config-notify

mgr/telemetry: fix waiting for mgr to warm up

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #43857 from aclamk/wip-aclamk-omap-clone-assert

os/bluestore: Protect _clone against sudden omap format changes

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

doc/releases/pacific.rst: add core updates for 16.2.7

16.2.7 fixes https://tracker.ceph.com/issues/53062, so remove the
"big scary warning" from the top of the pacific release page. We continue
to warn about this bug under the 16.2.6 section and in
https://docs.ceph.com/en/latest/releases/pacific/#upgrading-from-octopus-or-nautilus.

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc/releases/index.rst: change ref to 16.2.7

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc: 16.2.7 change log => 3 PRs added

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

doc: 16.2.7 Release Notes (dashboard)

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

doc: 16.2.7 Release Notes

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #44298 from adamemerson/wip-leveldb-release-note

doc: Add PendingReleaseNote for LevelDB removal

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/cephadm: agent: log response from mgr

Signed-off-by: Adam King <adking@redhat.com>

rgw:cleanup/refactor json and xml encoders and decoders

move the encoder and decoder methods into their associated class
files to eliminate undefined references to the class vtable

https://tracker.ceph.com/issues/53596

Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>

doc: Add PendingReleaseNote for LevelDB removal

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mgr/dashboard: disable Promql test in ARM

Temporarily disable this test while debugging the issue (since https://github.com/ceph/ceph/pull/43669
originally passed the ARM check).

Fixes: https://tracker.ceph.com/issues/53451
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #44009 from cbodley/wip-qa-cls-rgw-gc

qa/rgw: run ceph_test_cls_rgw_gc in rgw/verify suite

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

mailmap: Add Christopher Hoffman

Signed-off-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #42905 from sebastian-philipp/service_spec_no_redundant_placement

python-common: improve OSD spec error messages

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #44218 from guits/guits-issue-44356

ceph-volume: fix error 'KeyError' with inventory

Merge pull request #44219 from guits/guits-issue-53425

ceph-volume: fix tags dict output in `lvm list`

Merge pull request #44281 from athanatos/sjust/wip-53555

crimson/os/seastore: index lba pins atomically with addition to cache

Reviewed-by: Xuehan Xu <xuxuehan@360.cn>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: fix interruptor assert when no pg in peering_event

when no pg created, can't use interruptor

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

ceph.spec.in: fix mgr-cephadm CherryPy requirement for SUSE builds

Commit 78983ad0d0c added cherrypy to ceph-mgr-cephadm's Requires,
but this needs to be split out into distro-specific sections due
to subtle/irritating naming differences.

Fixes: 78983ad0d0cce422da32dc4876ac186f6d32c3f5
Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #44225 from liewegas/fix-53506

osd/OSDMapMapping: fix spurious threadpool timeout errors

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #44025 from ljflores/wip-remove-aggregated-perf-data

mgr/telemetry: remove aggregated perf metrics from the perf channel

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #43612 from adamemerson/wip-unleveling

build: Remove LevelDB support

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore/cache: init extents prior to read

Thus should ensure that any captured members of extent_init_func are
still valid at the cost of not being able to access the contents of the
extent at invocation time.  With this, we should be able to rely on any
logical extents/lba extents in the cache having validly initialized lba
pins.

Fixes: https://tracker.ceph.com/issues/53555
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seatore/.../lba_btree: update get_*_node to add_pin without reading node contents

This will allow us to do add_pin before we perform the actual extent read.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: pass depth/begin/end to get_*_node

We'll need this to populate the pin fields prior to read.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction_manager: clarify that init lambda only runs on new extents

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: remove unnecessary forward

Signed-off-by: Samuel Just <sjust@redhat.com>

python-common: add test inputs verifying count & count-per-host >= 1

This adds unit new test inputs, local to python-common that verify the
correct error messages are raised when count == 0 and count_per_host ==
0.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common: make count & count-per-host >= 1 checks consistent

The previous version of the validate function had a incorrect error
statement that suggested the count must be >1  when it should have
been >=1. This confusion was possibly due to using "n < 1" on
one line and "n <= 0" on another line. Since both values are supposed
to be integers this change corrects the error message and makes
the comparisons on the lines both use "n < 1" (since I find it easier
to see that the check "n < 1" is the inverse of the error text
asserting "n >= 1").

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common: add unit test func for invalid yaml inputs

I didn't find a preexisting test function for this so I added a
new test that is fed yaml snippets and expected error messages.
This verifies some of the recently added validation for
count and cound_per_host under the placement spec.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common: add int value validation for count and count_per_host

Add additional validation for the count and count_per_host fields
sourced from YAML.

Fixes: https://tracker.ceph.com/issues/50524
Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #44262 from cbodley/wip-53325

test/rgw: disable cls_rgw_gc test cases with defer_gc()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #44206 from linuxbox2/wip-rgwlc-9969

rgwlc:  remove magic debug blocks for clearing stale lc entries

Merge pull request #43991 from rhcs-dashboard/introduce-haroxy-metrics-rgw

mgr/dashboard: introduce HAProxy metrics for RGW

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43258 from soumyakoduri/wip-skoduri-dbstore-mp

rgw/dbstore: Multipart upload APIs

Merge pull request #42104 from linuxbox2/wip-rgwadminops-fsid

rgw: expose RADOS cluster_fsid via adminops

rgw:adminops: remove "import json" from radosgw_admin_rest.py

This is perhaps erring a bit on the side of cosmetic fixes.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #44220 from vshankar/wip-53487

qa: exclude `nofallback` mount option when using v1-style syntax

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/rgw: run ceph_test_cls_rgw_gc in rgw/verify suite

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: disable cls_rgw_gc test cases with defer_gc()

Fixes: https://tracker.ceph.com/issues/53325
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw:adminops: slightly generalize /info

Adds a get_name() method to rgw::sal::Store, by which each store
returns its unique name in lowercase.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw:sal: implement (placeholder) get_cluster_fsid in DBStore

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw:adminops: add test case for 'info' section

Add 'info' section test case to the radosgw_admin_test.py qa
task.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: introduce new /{admin}/info api resource

The new resource returns an array of informational data, which
currently consists of the RADOS cluster fsid as "cluster_fsid."

Fixes: https://tracker.ceph.com/issues/51432
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw:sal: expose cluster fsid in RGWStore

Adds a get_cluster_id() method to RGWStore, provided librados
cluster_fsid in the RADOS implementation.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #44001 from votdev/issue_53317_fix_proxy_conf

mgr/dashboard: API docs UI does not work with Angular dev server

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #44237 from pritha-srivastava/wip-rgw-abort-multipart-bypass-gc

rgw: deleting objects inline in case bypass_gc is specified

mgr/dashboard: introduce HAProxy metrics for RGW

Fixes: https://tracker.ceph.com/issues/53311
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

qa: exclude `nofallback` mount option when using v1-style syntax

Otherwise, certain upgrade tests fail which install pacific
or earlier releases since the mount helper does not understand
this mount option, thereby passing it to the kernel which would
does not handle this config causing mount to fail in tests.

Note that this mount config is only used during teuthology tests
to catch v2-style syntax implementation bugs in the kernel.

Fixes: http://tracker.ceph.com/issues/53487
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44127 from cyx1231st/wip-seastore-merge-journal-headers-2

crimson/os/seastore: make JournalSubmitter aware of header merging

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: make JournalSubmitter aware of header merging

Introduce an option seastore_journal_batch_preferred_fullness for
JournalSubmitter to control the preferred fullness to trigger batch
flushes.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: refactor get_encoded_length_after()

To return a more informative record_group_size_t.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: debug print journal write details

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: measure records write in journal

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: fix record metrics in cache

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: misc cleanup

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: add debug logs to print transaction name

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: don't print detail if the extent is not ready

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: print extent length

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: move implementation to seastore_types.cc

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>