common: Fix missing utility include

See:
    https://en.cppreference.com/w/cpp/utility/move

Detected on FreeBSD/Clang/libc++:
/home/jenkins/workspace/ceph-master-compile/src/common/deleter.h:111:43: error: no member named 'move' in namespace 'std'
  impl(deleter next) : refs(1), next(std::move(next)) {}

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #42750 from aclamk/wip-aclamk-bluefs-log-incremental

os/bluestore: incremental update mode for bluefs log

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #44368 from ceph/fix-api-tests

mgr/dashboard: run-backend-api-tests.sh: Older setuptools

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

os/bluestore/bluefs: Add tracking of bluefs log in noop replay mode

Keep updating bluefs log when printing content of bluefs replay log.
Without this modification we only have initial content of log.
Log can be printed by 'ceph-bluestore-tool bluefs-log-dump'.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

os/bluestore/bluefs: Sync BlueFS log with its allocation delta

BlueFS log is the only file that we can append to.
When we append to file we must take into consideration previously commited allocations,
otherwise update will be miscalculated.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

test/objectstore/bluefs_test: Add test for continuation of previous BlueFS log

Added test that verifies that in update mode we properly pick up delta.
BlueFS log is the only file that can be appended to, but it is done in very indirect way.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

os/bluefs: allow incremental file metadata updates in bluefs log

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge PR #44315 into master

* refs/pull/44315/head:
doc/cephfs: mds default cache memory limit is now 4GB

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

pybind/mgr/dashboard/run-backend-api-tests: Older setuptools

https://github.com/pypa/setuptools/issues/2353

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #42416 from galsalomon66/wip-test-albin-error-status

RGW/s3select : new-s3select-responds presto-alignments output-serialization

Merge pull request #44350 from Huber-ming/fix_annotation_misspelling

rgw: fix misspelling of 'effort'

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #44277 from fengchunsong/dpdk-rte-exit

msg/async/dpdk: replacing rte_exit with ceph_assert to avoid exit fai…

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

using fmt for message formatting

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

fix comments; enum-class; typo; documentation

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

typo

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

modification for error-handling and boto3

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

adding/modifying documentation for output-serialization, s3select-response, syntax support, error-handling

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

s3select: point to s3select/master(after merging pr #90)

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

in the case Presto is push-down query, it also send output-serialization parameter('&#10;'), that should be handle as '\n'

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

s3tests.yaml points to ceph-master

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

RGW: Implement continuation, progress, stats, end s3select response

    RGW/S3select: Implement output-serializationi. user may request different CSV defintions
    for output (field delimiter, row delimiter, quote handling.

    RGW/S3select: Implement presto-alignments. presto-application sends
    queries with table-alias,case insensitive, and with no-semicolon at the
    end of statement.

    RGW/s3select: zero object size issue

Signed-off-by: Albin Antony <aantony@redhat.com>
Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

rgw: fix misspelling of 'effort'

Signed-off-by: Huber-ming <zhangsm01@inspur.com>

Merge pull request #43139 from josephsawaya/rook-orch-qa

qa/tasks/rook: test reapplication of drive groups stored in mgr

Merge PR #44337 into master

* refs/pull/44337/head:
mon: prevent new sessions during shutdown

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #43872 from Huber-ming/reshard

rgw: init bucket index only if putting bucket instance info succeeds

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #43817 from joke-lee/UploadPartCopy-src-not-exist

rgw: fix UploadPartCopy error code when src object not exist and src bucket not exist

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #43259 from paulreece42/wip-rgw-xmlfix

rgw: allow CompletedMultipartUpload in XML

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41119 from Huber-ming/ls_mp

rgw: fix error ListMultipart allways returns StorageClass:STANDARD

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44336 from sebastian-philipp/cephadm-fix-test-facts

mgr/cephadm: Fix test_facts

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #44338 from rzarzynski/wip-crimson-abort_on_LOST_REVERT

crimson/osd: explicitly abort on LOST_REVERT in PG log.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44340 from rzarzynski/wip-crimson-backfill-discarding

crimson/osd: honor discarding old backfill messages

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44345 from xxhdx1985126/wip-onode-root-fix

crimson/os/seastore: reset onode in 'SeaStore::repeat_with_onode' before the transaction gets destroyed

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: reset onode in 'SeaStore::repeat_with_onode' before the transaction gets destroyed

Onodes hold references to the onode tree extents. And if it's referencing the root extent, that root
extent is cached in the onode trees root_tracker which caches onode tree roots by transaction address.
Than root_tracker entry only gets removed when the onode(or the corresponding "super") is destroyed.
On the other hand, two non-concurrent transactions can occupy the same address. So if an onode gets destroyed
after its transaction is destroyed, there will be a chance that another transaction occupying the same
address get that not-yet-destroyed and may-be-outdated onode.

BTW, Since we already cache extents in transactions, might want to drop onode tree root_tracker later?

Fixes: https://tracker.ceph.com/issues/53651
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/transaction: make sure the transaction's write_set is a subset of its read_set

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

src/vstart.sh: correct shell syntax error

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge PR #44228 into master

* refs/pull/44228/head:
qa/suites/orch/cephadm/osds: test 'ceph cephadm osd activate'
mgr/cephadm/services/osd: skip found osds that already have daemons
mgr/cephadm: allow activation of OSDs that have previously started

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

qa/tasks/rook: test reapplication of drive groups stored in mgr

This commit adds testing for the drive_group_loop in the Rook orchestrator
that reapplies drive groups that were applied previously.

This test removes an OSD, zaps the underlying device then waits for the OSD
to be re-created by the drive_group_loop.

This commit also updates the rook test suite to test v1.7.2 instead of 1.7.0
since `orch device zap` is only supported from v1.7.2 onwards.

Fixes: https://tracker.ceph.com/issues/53501
Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

qa/suites/orch/cephadm/osds: test 'ceph cephadm osd activate'

Make sure this command behaves when the /var/lib/ceph osd.NNN dir is
removed.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #42009 from pritha-srivastava/wip-rgw-sts-correct-read-obj-perm

rgw/sts: fix read_obj_policy permission evaluation

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #44331 from ronen-fr/wip-rf-tips187

crimson/osd: removing an unneeded make_unique()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement op discarding for backfill.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: generalize PG::old_peering_msg() to handle query_epoch.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: dissect the lpr check into PG::old_peering_msg().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #44280 from chrisphoffman/warn_pg

mon: Omit MANY_OBJECTS_PER_PG warning when autoscaler is on

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/osd: explicitly abort on LOST_REVERT in PG log.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #43901 from pcuzner/snmp-notifier

mgr/cephadm: Add snmp-gateway service support

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: Fix test_facts

Wasn't executed before

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm/services/osd: skip found osds that already have daemons

If we are trying to deploy new or newly-found osds, we can skip the ones
that already have cephadm daemons deployed.

Fixes: https://tracker.ceph.com/issues/53491
Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: allow activation of OSDs that have previously started

When this code was introduced way back in ea987a0e56db106f7c76d11f86b3e602257f365e,
for some reason I was focused only on freshly created OSDs.  The
get_osd_uuid_map() helper is used by deploy_osd_daemons_for_existing_osds()
which is called not only by OSD creation but also by 'ceph cephadm
osd activate', which is meant to instantiate daemons for existing OSD
devices (e.g., devices that were reattached to a new server, or whose
/var/lib/ceph/$fsid/osd.$id directory was lost for some other reason.
However, if we ignore OSDs with up_from > 0, then we can't recreate a
daemon instance for such existing OSDs--arguably the most important ones,
since they may hold real data.

Fixes: https://tracker.ceph.com/issues/53491
Signed-off-by: Sage Weil <sage@newdream.net>

mon: prevent new sessions during shutdown

From shutdown() we set STATE_SHUTDOWN and then call remove_all_sessions().
ms_handle_accept() is the only caller of add_session, so verifying that
we aren't shutting down (while under the session_map_lock) is sufficient
to prevent any new sessions from being added.

Fixes: https://tracker.ceph.com/issues/39150
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #44293 from sebastian-philipp/extract-uid-gid-readable

cephadm: make extract_uid_gid errors more readable

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #44208 into master

* refs/pull/44208/head:
mgr/progress: avoid inefficient dump of all pg stats

Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #44211 into master

* refs/pull/44211/head:
mon: increase mon_down_mkfs_grace to 2m

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: make PG::can_discard_replica_op() reusable for RecoveryBackend.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: removing an unneeded make_unique()

As the desired lifetime of the object matches the lifetime if
it is allocated on the stack, and as no ownership is transferred,
there is no point in using a unique_ptr here.

And see Google's guidance (https://abseil.io/tips/187),
under "Common Anti-Pattern: Avoiding &".

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #43889 from Daniel-Pivonka/cephadm-reg-cred

mgr/cephadm: store contianer registry credentials in config-key

Reviewed-by: Sage Weil <sage@newdream.net>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44031 from adk3798/agent-thrash

 mgr/cephadm: agent: simplify handling of agent reports

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44267 from phlogistonjohn/jjm-ceph-orch-yaml-issue206961

python-common: add int value validation for count and count_per_host

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #44269 from SUSE/wip-fix-mgr-cephadm-suse-cherrypy-requires

ceph.spec.in: fix mgr-cephadm CherryPy requirement for SUSE builds

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Adam King <adking@redhat.com>

mgr/cephadm: Add snmp-gateway service support

Add a new snmp-gateway service to provide a bridge between
Prometheus and an SNMP management platform. The gateway
service uses https://github.com/maxwo/snmp_notifier to provide
an SNMP v2c and SNMP V3 support.

The SNMP V3 support mandates at least authentication, and also
offers authentication and privacy (encryption).

Fixes: https://tracker.ceph.com/issues/52920
Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm: Add unit tests for snmp-gateway support

Adds tests to validate the deployed configuration given a known
input context, and check the parameters created based on input
various input scenarios

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #44197 from josephsawaya/zns-segment-manager

crimson/os/seastore: add ZNSSegmentManager

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44323 from cbodley/wip-53252

rgw/amqp: add default case to silence compiler warning

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

mgr/cephadm: agent: simplify handling of agent reports

Don't try to do extra things like checking other agents
or updating health checks when agents report. Rely on
serve loop for that

This also should help with thrashing we've been seeing

Signed-off-by: Adam King <adking@redhat.com>

rgw/amqp: add default case to silence compiler warning

Fixes: https://tracker.ceph.com/issues/53252
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44131 from ceph/wip-yuriw-release-16.2.7-master

doc: 16.2.7 Release Notes

Merge pull request #44136 from adk3798/agent-logging

mgr/cephadm: agent: log response from mgr

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44249 from DimitriPapadopoulos/codespell

doc,man: Typos found by codespell

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44309 from foadlind/fix_cephadm_doc

doc/cephadm/upgrade: correct example command

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc,man: typos found by codespell

Signed-off-by: Dimitri Papadopoulos <3234522+DimitriPapadopoulos@users.noreply.github.com>

doc/cephfs: mds default cache memory limit is now 4GB

MDS default cache memory limit is now 4GB.

Signed-off-by: wangxinyu <wangxinyu@inspur.com>

mgr/cephadm: Updated docs for snmp-gateway support

Updated docs to show snmp-gateway usage. docs provide
guidance on SNMP versions supported and show CLI and
yaml deployment examples.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm: provide initial snmp gateway support

This patch enables the cephadm binary
to deploy an SNMP gateway based on -
https://hub.docker.com/r/maxwo/snmp-notifier

Fixes: https://tracker.ceph.com/issues/52920
Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #44216 from locallocal/master

os/bluestore: don't need separate variable to mark hits when lookup oid.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #43770 from ifed01/wip-ifed-fix-53002

os/bluestore: avoid premature onode release.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #43412 from yaarith/devicehealth_fix_tz

mgr/devicehealth: fix missing timezone from time delta calculation

Reviewed-by: Laura Flores <lflores@redhat.com>

crimson/os/seastore: add ZNSSegmentManager

This commit adds the ZNSSegmentManager, so SeaStore can
interface with ZNS devices using the linux/blkzoned
library.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

doc/releases/pacific.rst: Instructions for notable change

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #44163 from rhcs-dashboard/fix-53451-master

mgr/dashboard: disable Promql test in ARM

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: p-se <NOT@FOUND>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>

Merge pull request #44295 from chrisphoffman/add-email-contact

mailmap: Add Christopher Hoffman

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44299 from kalebskeithley/dencode2

rgw:cleanup/refactor json and xml encoders and decoders

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

os/bluestore: avoid premature onode release.

This was observed when onode's removal is followed by reading
and the latter causes object release before the removal is finalized.
The root cause is an improper 'pinned' state assessment in Onode::get

More detailed overview is:
At some point Onode::get() might face the case when nref == 2 and pinned = true
which means parallel incomplete put is running on the onode - ref count is
decremented but pinned state is still unmodified (and even lock hasn't been
acquired yet).
This might finally result in two puts racing over the same onode with nref == 2
which finally results in a premature onode release:
  // nref =3, pinned = 1
  // Thread 1                   Thread 2
  //   o->put()                   o->get()
  //   --nref(n = 2, pinned=1)
  //                              nref++ (n=3, pinned = 1)
  //                              return
  //                              ...
  //                              o->put()
  //                              --nref(n = 2)
  //                              pinned = 0,
  //                              --nref(n = 1)
  //                              ocs->_unpin_and_rm(o) -> o->put()
  //                                ...
  //                                --nref(n = 0)
  //                                release o
  //  o->c->get_onode_cache()
  //  FAULT!
  //
The suggested fix is to introduce additional atomic counter tracking
running put() functions. And permit onode release when both regular
nref and put_nref are both equal to zero.

Fixes: https://tracker.ceph.com/issues/53002
Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #44029 from cbodley/wip-rgw-beast-header-limit

rgw/beast: add max_header_size option with 16k default, up from 4k

Reviewed-by: Mark Kogan <mkogan@redhat.com>

cephadm: make extract_uid_gid errors more readable

Avoid dumping a traceback

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm/upgrade: correct example command

Update the ceph version used in the example upgrade command to match the one mentioned in the text above it.

Signed-off-by: Foad Lind <foad.lind@citynetwork.eu>

Merge pull request #44235 from xxhdx1985126/wip-onode-omap-hint-optimization

crimson/os/seastore: avoid onode/omap laddr hint conflicts as much as possible

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

crimson/os/seastore: make onode data/metadata laddr space reservation configurable

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: randomize metadata laddr hints

This should prevent omap and xattr extent allocations from clumping near
the onode's hint.  Additionally, only generate them past the default
16MB object_data_handler reservation.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common: DONOT call crimson::get_logger() if NDEBUG is defined

Avoid debug related perf degredation

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #44141 from xxhdx1985126/wip-53409

crimson/os/seastore/segment_cleaner: correct available space calculation

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #44290 from liu-chunmei/crimson-fix-no-pg

crimson/osd: fix interruptor assert when no pg in peering_event

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: correct available space calculation

Current available space calculation is wrong, it just counts the space occupied
by extents, deltas and other stuff are not taken into account.

Fixes: https://tracker.ceph.com/issues/53409
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/common: redirect interruptible future debug output to default subsys

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/segment_cleaner: add perf metrics for better monitoring

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #44015 from liewegas/fix-44012

osd/PeeringState: separate history's pruub from pg's

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43864 from yaarith/fix-config-notify

mgr/telemetry: fix waiting for mgr to warm up

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #43857 from aclamk/wip-aclamk-omap-clone-assert

os/bluestore: Protect _clone against sudden omap format changes

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

doc/releases/pacific.rst: add core updates for 16.2.7

16.2.7 fixes https://tracker.ceph.com/issues/53062, so remove the
"big scary warning" from the top of the pacific release page. We continue
to warn about this bug under the 16.2.6 section and in
https://docs.ceph.com/en/latest/releases/pacific/#upgrading-from-octopus-or-nautilus.

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc/releases/index.rst: change ref to 16.2.7

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc: 16.2.7 change log => 3 PRs added

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>