crimson/osd: print message when customize parameters

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: allow seastar application to print its helps

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: --help-seastar does no longer belong to unknown_args

Now that app_template handles --help-seastar, and prints out all
app-level usages if --help is included in command line options. There is
no need to add a dedicated --help-seastar by ourselves and translate it
to --help.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #44317 from aclamk/aclamk-fix-bluefs-import

Fix ceph-bluestore-tool bluefs-import command

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #44328 from yaarith/gather_crashinfo-indexerror

mgr/telemetry: check that backtrace is not empty when redacting

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #44327 from yaarith/gather_device_report-indexerror

mgr/telemetry: catch also IndexError in gather_device_report()

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #44314 from locallocal/master

osd: remove unused variable legacy_infos_oid in the function PG::peek_map_epoch

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43686 from myoungwon/wip-ceph-dedup-tool-make-dedup-obj

tool: add  chunk/object-dedup command

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44190 from rhcs-dashboard/grafana-regex

monitoring/grafana: improve grafana unit tests variable substitution

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44362 from rhcs-dashboard/kcli-e2e-improvements

mgr/dashboard: cephadm e2e job: improvements

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #43838 from rhcs-dashboard/fix-53128-master

mgr/dashboard: Cluster Expansion - Review Section: fixes and improvements

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #43942 from rhcs-dashboard/host-drain-frontend

mgr/dashboard: Implement drain host functionality in dashboard

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Implement drain host functionality in dashboard

Fixes: https://tracker.ceph.com/issues/51587
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #42750 from aclamk/wip-aclamk-bluefs-log-incremental

os/bluestore: incremental update mode for bluefs log

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #44368 from ceph/fix-api-tests

mgr/dashboard: run-backend-api-tests.sh: Older setuptools

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

os/bluestore/bluefs: Add tracking of bluefs log in noop replay mode

Keep updating bluefs log when printing content of bluefs replay log.
Without this modification we only have initial content of log.
Log can be printed by 'ceph-bluestore-tool bluefs-log-dump'.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

os/bluestore/bluefs: Sync BlueFS log with its allocation delta

BlueFS log is the only file that we can append to.
When we append to file we must take into consideration previously commited allocations,
otherwise update will be miscalculated.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

test/objectstore/bluefs_test: Add test for continuation of previous BlueFS log

Added test that verifies that in update mode we properly pick up delta.
BlueFS log is the only file that can be appended to, but it is done in very indirect way.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

os/bluefs: allow incremental file metadata updates in bluefs log

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge PR #44315 into master

* refs/pull/44315/head:
doc/cephfs: mds default cache memory limit is now 4GB

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

pybind/mgr/dashboard/run-backend-api-tests: Older setuptools

https://github.com/pypa/setuptools/issues/2353

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #42416 from galsalomon66/wip-test-albin-error-status

RGW/s3select : new-s3select-responds presto-alignments output-serialization

mgr/dashboard: cephadm e2e job: improvements

- Avoid jenkins env. error: Input file /workdir/./src/pybind/mgr/dashboard/ci/cephadm/ceph_cluster.yml not found.Leaving...
- More meaningful log output.

Fixes: https://tracker.ceph.com/issues/53666
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

mgr/dashboard: Cluster Expansion - Review Section: fixes and improvements

Ensure "Storage capacity" keeps the "Description : Value" approach ("Number of devices: X" and "Raw Capacity: Y" in different lines).Correct issue with "host by services" host count

Fixes: https://tracker.ceph.com/issues/53128
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #44350 from Huber-ming/fix_annotation_misspelling

rgw: fix misspelling of 'effort'

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #44277 from fengchunsong/dpdk-rte-exit

msg/async/dpdk: replacing rte_exit with ceph_assert to avoid exit fai…

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

using fmt for message formatting

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

fix comments; enum-class; typo; documentation

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

typo

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

modification for error-handling and boto3

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

adding/modifying documentation for output-serialization, s3select-response, syntax support, error-handling

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

s3select: point to s3select/master(after merging pr #90)

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

in the case Presto is push-down query, it also send output-serialization parameter('&#10;'), that should be handle as '\n'

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

s3tests.yaml points to ceph-master

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

RGW: Implement continuation, progress, stats, end s3select response

    RGW/S3select: Implement output-serializationi. user may request different CSV defintions
    for output (field delimiter, row delimiter, quote handling.

    RGW/S3select: Implement presto-alignments. presto-application sends
    queries with table-alias,case insensitive, and with no-semicolon at the
    end of statement.

    RGW/s3select: zero object size issue

Signed-off-by: Albin Antony <aantony@redhat.com>
Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

rgw: fix misspelling of 'effort'

Signed-off-by: Huber-ming <zhangsm01@inspur.com>

Merge pull request #43139 from josephsawaya/rook-orch-qa

qa/tasks/rook: test reapplication of drive groups stored in mgr

Merge PR #44337 into master

* refs/pull/44337/head:
mon: prevent new sessions during shutdown

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #43872 from Huber-ming/reshard

rgw: init bucket index only if putting bucket instance info succeeds

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #43817 from joke-lee/UploadPartCopy-src-not-exist

rgw: fix UploadPartCopy error code when src object not exist and src bucket not exist

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #43259 from paulreece42/wip-rgw-xmlfix

rgw: allow CompletedMultipartUpload in XML

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41119 from Huber-ming/ls_mp

rgw: fix error ListMultipart allways returns StorageClass:STANDARD

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44336 from sebastian-philipp/cephadm-fix-test-facts

mgr/cephadm: Fix test_facts

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #44338 from rzarzynski/wip-crimson-abort_on_LOST_REVERT

crimson/osd: explicitly abort on LOST_REVERT in PG log.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44340 from rzarzynski/wip-crimson-backfill-discarding

crimson/osd: honor discarding old backfill messages

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44345 from xxhdx1985126/wip-onode-root-fix

crimson/os/seastore: reset onode in 'SeaStore::repeat_with_onode' before the transaction gets destroyed

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: reset onode in 'SeaStore::repeat_with_onode' before the transaction gets destroyed

Onodes hold references to the onode tree extents. And if it's referencing the root extent, that root
extent is cached in the onode trees root_tracker which caches onode tree roots by transaction address.
Than root_tracker entry only gets removed when the onode(or the corresponding "super") is destroyed.
On the other hand, two non-concurrent transactions can occupy the same address. So if an onode gets destroyed
after its transaction is destroyed, there will be a chance that another transaction occupying the same
address get that not-yet-destroyed and may-be-outdated onode.

BTW, Since we already cache extents in transactions, might want to drop onode tree root_tracker later?

Fixes: https://tracker.ceph.com/issues/53651
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/transaction: make sure the transaction's write_set is a subset of its read_set

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

src/vstart.sh: correct shell syntax error

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge PR #44228 into master

* refs/pull/44228/head:
qa/suites/orch/cephadm/osds: test 'ceph cephadm osd activate'
mgr/cephadm/services/osd: skip found osds that already have daemons
mgr/cephadm: allow activation of OSDs that have previously started

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

qa/tasks/rook: test reapplication of drive groups stored in mgr

This commit adds testing for the drive_group_loop in the Rook orchestrator
that reapplies drive groups that were applied previously.

This test removes an OSD, zaps the underlying device then waits for the OSD
to be re-created by the drive_group_loop.

This commit also updates the rook test suite to test v1.7.2 instead of 1.7.0
since `orch device zap` is only supported from v1.7.2 onwards.

Fixes: https://tracker.ceph.com/issues/53501
Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

qa/suites/orch/cephadm/osds: test 'ceph cephadm osd activate'

Make sure this command behaves when the /var/lib/ceph osd.NNN dir is
removed.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #42009 from pritha-srivastava/wip-rgw-sts-correct-read-obj-perm

rgw/sts: fix read_obj_policy permission evaluation

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #44331 from ronen-fr/wip-rf-tips187

crimson/osd: removing an unneeded make_unique()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement op discarding for backfill.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: generalize PG::old_peering_msg() to handle query_epoch.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: dissect the lpr check into PG::old_peering_msg().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #44280 from chrisphoffman/warn_pg

mon: Omit MANY_OBJECTS_PER_PG warning when autoscaler is on

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/osd: explicitly abort on LOST_REVERT in PG log.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #43901 from pcuzner/snmp-notifier

mgr/cephadm: Add snmp-gateway service support

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: Fix test_facts

Wasn't executed before

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm/services/osd: skip found osds that already have daemons

If we are trying to deploy new or newly-found osds, we can skip the ones
that already have cephadm daemons deployed.

Fixes: https://tracker.ceph.com/issues/53491
Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: allow activation of OSDs that have previously started

When this code was introduced way back in ea987a0e56db106f7c76d11f86b3e602257f365e,
for some reason I was focused only on freshly created OSDs.  The
get_osd_uuid_map() helper is used by deploy_osd_daemons_for_existing_osds()
which is called not only by OSD creation but also by 'ceph cephadm
osd activate', which is meant to instantiate daemons for existing OSD
devices (e.g., devices that were reattached to a new server, or whose
/var/lib/ceph/$fsid/osd.$id directory was lost for some other reason.
However, if we ignore OSDs with up_from > 0, then we can't recreate a
daemon instance for such existing OSDs--arguably the most important ones,
since they may hold real data.

Fixes: https://tracker.ceph.com/issues/53491
Signed-off-by: Sage Weil <sage@newdream.net>

mon: prevent new sessions during shutdown

From shutdown() we set STATE_SHUTDOWN and then call remove_all_sessions().
ms_handle_accept() is the only caller of add_session, so verifying that
we aren't shutting down (while under the session_map_lock) is sufficient
to prevent any new sessions from being added.

Fixes: https://tracker.ceph.com/issues/39150
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #44293 from sebastian-philipp/extract-uid-gid-readable

cephadm: make extract_uid_gid errors more readable

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #44208 into master

* refs/pull/44208/head:
mgr/progress: avoid inefficient dump of all pg stats

Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #44211 into master

* refs/pull/44211/head:
mon: increase mon_down_mkfs_grace to 2m

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: make PG::can_discard_replica_op() reusable for RecoveryBackend.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: removing an unneeded make_unique()

As the desired lifetime of the object matches the lifetime if
it is allocated on the stack, and as no ownership is transferred,
there is no point in using a unique_ptr here.

And see Google's guidance (https://abseil.io/tips/187),
under "Common Anti-Pattern: Avoiding &".

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #43889 from Daniel-Pivonka/cephadm-reg-cred

mgr/cephadm: store contianer registry credentials in config-key

Reviewed-by: Sage Weil <sage@newdream.net>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44031 from adk3798/agent-thrash

 mgr/cephadm: agent: simplify handling of agent reports

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44267 from phlogistonjohn/jjm-ceph-orch-yaml-issue206961

python-common: add int value validation for count and count_per_host

Reviewed-by: Adam King <adking@redhat.com>

os/bluestore/ceph-bluestore-tool: Fix bluefs-import command

Modify bluefs-import command so it can properly initialize allocators.
Without allocators initialized, importing file to bluefs did overwrite some random data,
including first block on device.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #44269 from SUSE/wip-fix-mgr-cephadm-suse-cherrypy-requires

ceph.spec.in: fix mgr-cephadm CherryPy requirement for SUSE builds

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Adam King <adking@redhat.com>

mgr/cephadm: Add snmp-gateway service support

Add a new snmp-gateway service to provide a bridge between
Prometheus and an SNMP management platform. The gateway
service uses https://github.com/maxwo/snmp_notifier to provide
an SNMP v2c and SNMP V3 support.

The SNMP V3 support mandates at least authentication, and also
offers authentication and privacy (encryption).

Fixes: https://tracker.ceph.com/issues/52920
Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm: Add unit tests for snmp-gateway support

Adds tests to validate the deployed configuration given a known
input context, and check the parameters created based on input
various input scenarios

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/telemetry: check that backtrace is not empty when redacting

Some python crashes contain empty backtraces; check before redacting
their final line.

Fixes: https://tracker.ceph.com/issues/53604
Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>

Merge pull request #44197 from josephsawaya/zns-segment-manager

crimson/os/seastore: add ZNSSegmentManager

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/telemetry: catch also IndexError in gather_device_report()

When generating the device report, we obfuscate host names, which
exist in a device's 'location' key. Some devices do not have a
'location' key, thus we catch a KeyError in these cases; in other cases
the key exists, but its value is an empty list. Skip these too by
catching an IndexError.

Fixes: https://tracker.ceph.com/issues/53603
Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>

Merge pull request #44323 from cbodley/wip-53252

rgw/amqp: add default case to silence compiler warning

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

mgr/cephadm: agent: simplify handling of agent reports

Don't try to do extra things like checking other agents
or updating health checks when agents report. Rely on
serve loop for that

This also should help with thrashing we've been seeing

Signed-off-by: Adam King <adking@redhat.com>

rgw/amqp: add default case to silence compiler warning

Fixes: https://tracker.ceph.com/issues/53252
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44131 from ceph/wip-yuriw-release-16.2.7-master

doc: 16.2.7 Release Notes

Merge pull request #44136 from adk3798/agent-logging

mgr/cephadm: agent: log response from mgr

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44249 from DimitriPapadopoulos/codespell

doc,man: Typos found by codespell

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44309 from foadlind/fix_cephadm_doc

doc/cephadm/upgrade: correct example command

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc,man: typos found by codespell

Signed-off-by: Dimitri Papadopoulos <3234522+DimitriPapadopoulos@users.noreply.github.com>

monitoring/grafana: doctest util regex

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>

doc/cephfs: mds default cache memory limit is now 4GB

MDS default cache memory limit is now 4GB.

Signed-off-by: wangxinyu <wangxinyu@inspur.com>

osd: remove unused variable legacy_infos_oid in the function PG::peek_map_epoch.

Signed-off-by: locallocal <locallocal@163.com>

mgr/cephadm: Updated docs for snmp-gateway support

Updated docs to show snmp-gateway usage. docs provide
guidance on SNMP versions supported and show CLI and
yaml deployment examples.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm: provide initial snmp gateway support

This patch enables the cephadm binary
to deploy an SNMP gateway based on -
https://hub.docker.com/r/maxwo/snmp-notifier

Fixes: https://tracker.ceph.com/issues/52920
Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #44216 from locallocal/master

os/bluestore: don't need separate variable to mark hits when lookup oid.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #43770 from ifed01/wip-ifed-fix-53002

os/bluestore: avoid premature onode release.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #43412 from yaarith/devicehealth_fix_tz

mgr/devicehealth: fix missing timezone from time delta calculation

Reviewed-by: Laura Flores <lflores@redhat.com>

crimson/os/seastore: add ZNSSegmentManager

This commit adds the ZNSSegmentManager, so SeaStore can
interface with ZNS devices using the linux/blkzoned
library.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

doc/releases/pacific.rst: Instructions for notable change

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #44163 from rhcs-dashboard/fix-53451-master

mgr/dashboard: disable Promql test in ARM

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: p-se <NOT@FOUND>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>

Merge pull request #44295 from chrisphoffman/add-email-contact

mailmap: Add Christopher Hoffman

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44299 from kalebskeithley/dencode2

rgw:cleanup/refactor json and xml encoders and decoders

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>