mgr/telemetry: check that backtrace is not empty when redacting

Some python crashes contain empty backtraces; check before redacting
their final line.

Fixes: https://tracker.ceph.com/issues/53604
Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>

Merge pull request #44197 from josephsawaya/zns-segment-manager

crimson/os/seastore: add ZNSSegmentManager

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44323 from cbodley/wip-53252

rgw/amqp: add default case to silence compiler warning

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

rgw/amqp: add default case to silence compiler warning

Fixes: https://tracker.ceph.com/issues/53252
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44131 from ceph/wip-yuriw-release-16.2.7-master

doc: 16.2.7 Release Notes

Merge pull request #44136 from adk3798/agent-logging

mgr/cephadm: agent: log response from mgr

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44249 from DimitriPapadopoulos/codespell

doc,man: Typos found by codespell

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44309 from foadlind/fix_cephadm_doc

doc/cephadm/upgrade: correct example command

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc,man: typos found by codespell

Signed-off-by: Dimitri Papadopoulos <3234522+DimitriPapadopoulos@users.noreply.github.com>

Merge pull request #44216 from locallocal/master

os/bluestore: don't need separate variable to mark hits when lookup oid.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #43770 from ifed01/wip-ifed-fix-53002

os/bluestore: avoid premature onode release.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #43412 from yaarith/devicehealth_fix_tz

mgr/devicehealth: fix missing timezone from time delta calculation

Reviewed-by: Laura Flores <lflores@redhat.com>

crimson/os/seastore: add ZNSSegmentManager

This commit adds the ZNSSegmentManager, so SeaStore can
interface with ZNS devices using the linux/blkzoned
library.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

doc/releases/pacific.rst: Instructions for notable change

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #44163 from rhcs-dashboard/fix-53451-master

mgr/dashboard: disable Promql test in ARM

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: p-se <NOT@FOUND>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>

Merge pull request #44295 from chrisphoffman/add-email-contact

mailmap: Add Christopher Hoffman

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44299 from kalebskeithley/dencode2

rgw:cleanup/refactor json and xml encoders and decoders

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

os/bluestore: avoid premature onode release.

This was observed when onode's removal is followed by reading
and the latter causes object release before the removal is finalized.
The root cause is an improper 'pinned' state assessment in Onode::get

More detailed overview is:
At some point Onode::get() might face the case when nref == 2 and pinned = true
which means parallel incomplete put is running on the onode - ref count is
decremented but pinned state is still unmodified (and even lock hasn't been
acquired yet).
This might finally result in two puts racing over the same onode with nref == 2
which finally results in a premature onode release:
  // nref =3, pinned = 1
  // Thread 1                   Thread 2
  //   o->put()                   o->get()
  //   --nref(n = 2, pinned=1)
  //                              nref++ (n=3, pinned = 1)
  //                              return
  //                              ...
  //                              o->put()
  //                              --nref(n = 2)
  //                              pinned = 0,
  //                              --nref(n = 1)
  //                              ocs->_unpin_and_rm(o) -> o->put()
  //                                ...
  //                                --nref(n = 0)
  //                                release o
  //  o->c->get_onode_cache()
  //  FAULT!
  //
The suggested fix is to introduce additional atomic counter tracking
running put() functions. And permit onode release when both regular
nref and put_nref are both equal to zero.

Fixes: https://tracker.ceph.com/issues/53002
Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #44029 from cbodley/wip-rgw-beast-header-limit

rgw/beast: add max_header_size option with 16k default, up from 4k

Reviewed-by: Mark Kogan <mkogan@redhat.com>

doc/cephadm/upgrade: correct example command

Update the ceph version used in the example upgrade command to match the one mentioned in the text above it.

Signed-off-by: Foad Lind <foad.lind@citynetwork.eu>

Merge pull request #44235 from xxhdx1985126/wip-onode-omap-hint-optimization

crimson/os/seastore: avoid onode/omap laddr hint conflicts as much as possible

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

crimson/os/seastore: make onode data/metadata laddr space reservation configurable

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: randomize metadata laddr hints

This should prevent omap and xattr extent allocations from clumping near
the onode's hint.  Additionally, only generate them past the default
16MB object_data_handler reservation.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common: DONOT call crimson::get_logger() if NDEBUG is defined

Avoid debug related perf degredation

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #44141 from xxhdx1985126/wip-53409

crimson/os/seastore/segment_cleaner: correct available space calculation

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #44290 from liu-chunmei/crimson-fix-no-pg

crimson/osd: fix interruptor assert when no pg in peering_event

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: correct available space calculation

Current available space calculation is wrong, it just counts the space occupied
by extents, deltas and other stuff are not taken into account.

Fixes: https://tracker.ceph.com/issues/53409
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/common: redirect interruptible future debug output to default subsys

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/segment_cleaner: add perf metrics for better monitoring

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #44015 from liewegas/fix-44012

osd/PeeringState: separate history's pruub from pg's

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43864 from yaarith/fix-config-notify

mgr/telemetry: fix waiting for mgr to warm up

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #43857 from aclamk/wip-aclamk-omap-clone-assert

os/bluestore: Protect _clone against sudden omap format changes

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

doc/releases/pacific.rst: add core updates for 16.2.7

16.2.7 fixes https://tracker.ceph.com/issues/53062, so remove the
"big scary warning" from the top of the pacific release page. We continue
to warn about this bug under the 16.2.6 section and in
https://docs.ceph.com/en/latest/releases/pacific/#upgrading-from-octopus-or-nautilus.

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc/releases/index.rst: change ref to 16.2.7

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc: 16.2.7 change log => 3 PRs added

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

doc: 16.2.7 Release Notes (dashboard)

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

doc: 16.2.7 Release Notes

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #44298 from adamemerson/wip-leveldb-release-note

doc: Add PendingReleaseNote for LevelDB removal

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/cephadm: agent: log response from mgr

Signed-off-by: Adam King <adking@redhat.com>

rgw:cleanup/refactor json and xml encoders and decoders

move the encoder and decoder methods into their associated class
files to eliminate undefined references to the class vtable

https://tracker.ceph.com/issues/53596

Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>

doc: Add PendingReleaseNote for LevelDB removal

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mgr/dashboard: disable Promql test in ARM

Temporarily disable this test while debugging the issue (since https://github.com/ceph/ceph/pull/43669
originally passed the ARM check).

Fixes: https://tracker.ceph.com/issues/53451
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #44009 from cbodley/wip-qa-cls-rgw-gc

qa/rgw: run ceph_test_cls_rgw_gc in rgw/verify suite

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

mailmap: Add Christopher Hoffman

Signed-off-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #42905 from sebastian-philipp/service_spec_no_redundant_placement

python-common: improve OSD spec error messages

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #44218 from guits/guits-issue-44356

ceph-volume: fix error 'KeyError' with inventory

Merge pull request #44219 from guits/guits-issue-53425

ceph-volume: fix tags dict output in `lvm list`

Merge pull request #44281 from athanatos/sjust/wip-53555

crimson/os/seastore: index lba pins atomically with addition to cache

Reviewed-by: Xuehan Xu <xuxuehan@360.cn>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: fix interruptor assert when no pg in peering_event

when no pg created, can't use interruptor

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #44225 from liewegas/fix-53506

osd/OSDMapMapping: fix spurious threadpool timeout errors

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #44025 from ljflores/wip-remove-aggregated-perf-data

mgr/telemetry: remove aggregated perf metrics from the perf channel

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #43612 from adamemerson/wip-unleveling

build: Remove LevelDB support

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore/cache: init extents prior to read

Thus should ensure that any captured members of extent_init_func are
still valid at the cost of not being able to access the contents of the
extent at invocation time.  With this, we should be able to rely on any
logical extents/lba extents in the cache having validly initialized lba
pins.

Fixes: https://tracker.ceph.com/issues/53555
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seatore/.../lba_btree: update get_*_node to add_pin without reading node contents

This will allow us to do add_pin before we perform the actual extent read.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: pass depth/begin/end to get_*_node

We'll need this to populate the pin fields prior to read.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction_manager: clarify that init lambda only runs on new extents

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: remove unnecessary forward

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #44262 from cbodley/wip-53325

test/rgw: disable cls_rgw_gc test cases with defer_gc()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #44206 from linuxbox2/wip-rgwlc-9969

rgwlc:  remove magic debug blocks for clearing stale lc entries

Merge pull request #43991 from rhcs-dashboard/introduce-haroxy-metrics-rgw

mgr/dashboard: introduce HAProxy metrics for RGW

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43258 from soumyakoduri/wip-skoduri-dbstore-mp

rgw/dbstore: Multipart upload APIs

Merge pull request #42104 from linuxbox2/wip-rgwadminops-fsid

rgw: expose RADOS cluster_fsid via adminops

rgw:adminops: remove "import json" from radosgw_admin_rest.py

This is perhaps erring a bit on the side of cosmetic fixes.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #44220 from vshankar/wip-53487

qa: exclude `nofallback` mount option when using v1-style syntax

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/rgw: run ceph_test_cls_rgw_gc in rgw/verify suite

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: disable cls_rgw_gc test cases with defer_gc()

Fixes: https://tracker.ceph.com/issues/53325
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw:adminops: slightly generalize /info

Adds a get_name() method to rgw::sal::Store, by which each store
returns its unique name in lowercase.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw:sal: implement (placeholder) get_cluster_fsid in DBStore

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw:adminops: add test case for 'info' section

Add 'info' section test case to the radosgw_admin_test.py qa
task.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: introduce new /{admin}/info api resource

The new resource returns an array of informational data, which
currently consists of the RADOS cluster fsid as "cluster_fsid."

Fixes: https://tracker.ceph.com/issues/51432
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw:sal: expose cluster fsid in RGWStore

Adds a get_cluster_id() method to RGWStore, provided librados
cluster_fsid in the RADOS implementation.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #44001 from votdev/issue_53317_fix_proxy_conf

mgr/dashboard: API docs UI does not work with Angular dev server

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #44237 from pritha-srivastava/wip-rgw-abort-multipart-bypass-gc

rgw: deleting objects inline in case bypass_gc is specified

mgr/dashboard: introduce HAProxy metrics for RGW

Fixes: https://tracker.ceph.com/issues/53311
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

qa: exclude `nofallback` mount option when using v1-style syntax

Otherwise, certain upgrade tests fail which install pacific
or earlier releases since the mount helper does not understand
this mount option, thereby passing it to the kernel which would
does not handle this config causing mount to fail in tests.

Note that this mount config is only used during teuthology tests
to catch v2-style syntax implementation bugs in the kernel.

Fixes: http://tracker.ceph.com/issues/53487
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44127 from cyx1231st/wip-seastore-merge-journal-headers-2

crimson/os/seastore: make JournalSubmitter aware of header merging

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: make JournalSubmitter aware of header merging

Introduce an option seastore_journal_batch_preferred_fullness for
JournalSubmitter to control the preferred fullness to trigger batch
flushes.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: refactor get_encoded_length_after()

To return a more informative record_group_size_t.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: debug print journal write details

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: measure records write in journal

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: fix record metrics in cache

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: misc cleanup

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: add debug logs to print transaction name

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: don't print detail if the extent is not ready

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: print extent length

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: move implementation to seastore_types.cc

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: merge records metadata if they are grouped

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: mark out empty transactions

TODO: avoid write if the transaction is empty.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: refactor, introduce record_t and record_group_t with sizes

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: scan records based on record_locator_t

Record may not have its own base if headers are merged.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: add more checks when read record_header_t

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: misc cleanup and reformat

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: add logs in ExtentReader

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #43919 from ronen-fr/wip-rf-test-nodeep

osd/scrub (& qa/standalone): test for scrub behavior when no-scrub is set but no-deep-scrub is not

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #43305 from heylinn/ceph_rundir_sysvinit

init-ceph: create /var/run/ceph for sysvinit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #44090 from sseshasa/wip-fix-require-osd-release

osd/OSDMap: Add health warning if 'require-osd-release' != current release

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #44244 from cyx1231st/wip-seastore-refine-metrics

crimson/os/seastore: refine transaction metrics

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44242 from liu-chunmei/crimson-fix-heartbeat-addrs

crimson/osd: fix heartbeat front and back blank ip

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44098 from ifed01/wip-ifed-dump-alloc-unit

os/bluestore: dump bluestore/bluefs alloc unit sizes with perf dump

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #43840 from ifed01/wip-ifed-verbose-open-col

osd,bluestore: gracefully handle a failure during meta collection load

Reviewed-by: jdurgin@redhat.com
Reviewed-by: nojha@redhat.com