crush: cancel upmaps with up set size > pool size

Fixes: https://tracker.ceph.com/issues/51842
Signed-off-by: huangjun <huangjun@xsky.com>

Merge pull request #42247 from pritha-srivastava/wip-rgw-sts-session-policy-fixes

rgw/sts: createbucket op should take session_policies into account

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #42249 from kalebskeithley/rgw-51599

rgw: cleanup virtual dtor decls and defns

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42365 from PracticalMetal/typo

doc: Fix typos and grammar in ecbackend.rst

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #42431 into master

* refs/pull/42431/head:
cmake: add "mypy" back to tox envlist of "qa""
qa/tasks/vstart_runner: add optional "sudo" param to _run_python()

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #42444 into master

* refs/pull/42444/head:
doc/cephfs/journaler: document options using confval directive

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42380 from adamemerson/wip-51712

rgw: radosgw-admin errors if marker not specified on data/mdlog trim

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42329 from vshankar/wip-cephfs-mirror-dir-remove-registery

cephfs-mirror: record directory path cancel in DirRegistry

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #42450 from tchaikov/wip-debian-jaeger

debian/control: depend on libjaeger only if <pkg.ceph.jaeger>

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #42433 from sebastian-philipp/fix-gen-keepalive-config

mgr/cephadm: Fix haproxy not being recognized as a proper daemon

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Asbjørn Sannes <asbjorn.sannes@interhost.no>

debian/control: depend on libjaeger only if <pkg.ceph.jaeger>

otherwise we'd have following failure when trying to install ceph
packages if they are not built with pkg.ceph.jaeger profile:

ceph-common : Depends: libjaeger (= 17.0.0-6321-g62349ba4-1focal) but it is not installable

this change should address the regression introduced by
4b69d3a1844433845eb7000f56be72fb11f4e737

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: Fix haproxy not being recognized as a proper daemon

Turns out daemon types != service types:

    cephadm [WRN] Found unknown service type haproxy on host smithi019
    cephadm [WRN] Found unknown service type keepalived on host smithi019

leading to `self.mgr.cache.get_daemons_by_service(spec.service_name())`
not returning any daemons.

Fixes: https://tracker.ceph.com/issues/51311
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc: fix typo and grammar

Signed-off-by: Pulkit Mittal <2pulkit2@gmail.com>

Merge pull request #38783 from ideepika/wip-osd-tracing

cmake/ninja: support ninja for jaegertracing

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: fix jaeger linking to ceph libraries

updates:
* custom target jaeger_base
* target jaeger-base which encapsulates all jaeger libs
* include external libraries path needed linking libraries and including
  opentracing and jaegertracing headers files

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake: change debian DESTDIR for installing jaeger deps

debian uses debian/tmp as destination dir for installing build files,
but since we are using common path($build_dir/external) available both
for rpm and debian based dependency installation, it becomes far more
complicated to maintain include/link path for these external projects.

elaborating on it:
path we are configuring for both rpm and debian installing including,
and linking of external librarires:
/build/ceph-17.0.0-5779-g928f9e55/obj-x86_64-linux-gnu/external/
debian appends DESTDIR to this path, and hence our predefined target
artificats cannot find correct path for external libs, I tried adding
ENV${DESTDIR} so that it could include correct external lib install
path, but it still cannot find them:

failed to link in case of:
-  install(DIRECTORY $ENV{DESTDIR}${CMAKE_BINARY_DIR}/external/include/jaegertracing
-                $ENV{DESTDIR}${CMAKE_BINARY_DIR}/external/include/opentracing
-          DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
-  include_directories(SYSTEM ${CMAKE_INSTALL_INCLUDEDIR}/jaegertracing)
-  include_directories(SYSTEM ${CMAKE_INSTALL_INCLUDEDIR}/opentracing)

-- Installing: /build/ceph-17.0.0-5790-g6bc03cbd/debian/tmp/build/ceph-17.0.0-5790-g6bc03cbd/obj-x86_64-linux-gnu/external/include/jaegertracing/Tracer.h

cd /build/ceph-17.0.0-5790-g6bc03cbd/obj-x86_64-linux-gnu/src && /usr/bin/c++  -DBOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION -DBOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT -DHAVE_CONFIG_H -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE -D__CEPH__ -D__STDC_FORMAT_MACROS -D__linux__ -I/build/ceph-17.0.0-5790-g6bc03cbd/obj-x86_64-linux-gnu/src/include -I/build/ceph-17.0.0-5790-g6bc03cbd/src -isystem /build/ceph-17.0.0-5790-g6bc03cbd/obj-x86_64-linux-gnu/boost/include -isystem /build/ceph-17.0.0-5790-g6bc03cbd/obj-x86_64-linux-gnu/include -isystem /build/ceph-17.0.0-5790-g6bc03cbd/src/xxHash -isystem /build/ceph-17.0.0-5790-g6bc03cbd/src/rapidjson/include -isystem /build/ceph-17.0.0-5790-g6bc03cbd/src/include/jaegertracing -isystem /build/ceph-17.0.0-5790-g6bc03cbd/src/include/opentracing  -g -O2 -fdebug-prefix-map=/build/ceph-17.0.0-5790-g6bc03cbd=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC   -U_FORTIFY_SOURCE -Wall -fno-strict-aliasing -fsigned-char -Wtype-limits -Wignored-qualifiers -Wpointer-arith -Werror=format-security -Winit-self -Wno-unknown-pragmas -Wnon-virtual-dtor -Wno-ignored-qualifiers -ftemplate-depth-1024 -Wpessimizing-move -Wredundant-move -Wstrict-null-sentinel -Woverloaded-virtual -fno-new-ttp-matching -fstack-protector-strong -fdiagnostics-color=auto -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -std=c++17 -o CMakeFiles/common-objs.dir/mds/mdstypes.cc.o -c /build/ceph-17.0.0-5790-g6bc03cbd/src/mds/mdstypes.cc
In file included from /build/ceph-17.0.0-5790-g6bc03cbd/src/osd/OpRequest.h:21,
                 from /build/ceph-17.0.0-5790-g6bc03cbd/src/osd/OpRequest.cc:3:
/build/ceph-17.0.0-5790-g6bc03cbd/src/common/tracer.h:10:10: fatal error: jaegertracing/Tracer.h: No such file or directory

Since the install path is in our build environment for these librarires,
skipping DESTDIR looks to me hacky fix, but does the job.
with empty destdir:

-- Installing: /build/ceph-17.0.0-5791-gb97b9640/obj-x86_64-linux-gnu/external/include/jaegertracing/Tracer.h

cd /build/ceph-17.0.0-5791-gb97b9640/obj-x86_64-linux-gnu/src/mon && /usr/bin/c++  -DBOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION -DBOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT -DHAVE_CONFIG_H -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE -D__CEPH__ -D__STDC_FORMAT_MACROS -D__linux__ -I/build/ceph-17.0.0-5791-gb97b9640/obj-x86_64-linux-gnu/src/include -I/build/ceph-17.0.0-5791-gb97b9640/src -isystem /build/ceph-17.0.0-5791-gb97b9640/obj-x86_64-linux-gnu/boost/include -isystem /build/ceph-17.0.0-5791-gb97b9640/obj-x86_64-linux-gnu/include -isystem /build/ceph-17.0.0-5791-gb97b9640/src/xxHash -isystem /build/ceph-17.0.0-5791-gb97b9640/src/rapidjson/include -isystem /build/ceph-17.0.0-5791-gb97b9640/obj-x86_64-linux-gnu/external/include -isystem /build/ceph-17.0.0-5791-gb97b9640/src/rocksdb/include  -g -O2 -fdebug-prefix-map=/build/ceph-17.0.0-5791-gb97b9640=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC   -U_FORTIFY_SOURCE -Wall -fno-strict-aliasing -fsigned-char -Wtype-limits -Wignored-qualifiers -Wpointer-arith -Werror=format-security -Winit-self -Wno-unknown-pragmas -Wnon-virtual-dtor -Wno-ignored-qualifiers -ftemplate-depth-1024 -Wpessimizing-move -Wredundant-move -Wstrict-null-sentinel -Woverloaded-virtual -fno-new-ttp-matching -fstack-protector-strong -fdiagnostics-color=auto -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -std=c++17 -o CMakeFiles/mon.dir/MgrMonitor.cc.o -c /build/ceph-17.0.0-5791-gb97b9640/src/mon/MgrMonitor.cc

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake: adds BUILD_BYPRODUCT for external build jaegertracing libs

* adds BUILD_BYPRODUCT which tells ninja which library will be generated
after the build(needed for dependent build libs)

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake: minor reorder of boost path in jaeger build external projet

* Boost is a dependency for jaeger, to use the right version, we pass
ceph build boost path to cmake jaeger build step

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake: target name Jaeger >> jaegertracing

since jaegertracing is the original target that jaeger submodule uses in
it's cmake, cmake build complained if named otherwise

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake: move ExternalProjectHelper & IncludeJaeger to BuildJaeger

set_library_properties_for_external_project assists with setting right target
properties for all jaeger dependencies.
IncludeJaeger would take care of linking and creating these targets
having them spread out when they are highly coupled seems not optimal.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

debian, install-deps.sh: use Build-Profiles to optionally build with jaeger

we will now be using BuildProfileSpec based optional pkg building,
removing comment from .install file is no longer needed If
pkg.ceph.jaeger is enabled debian/control, it shall work to install
jaeger and it's dependencies.

  https://wiki.debian.org/BuildProfileSpec.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake: add find_package(thrift) support for thrift 0.13

adds Findthrift.cmake which is used to find thrift 0.13, as a dependency
for building jaegertracing

* bump up submodule version for jaeger-client-cpp for thrift compiler
removal

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

ceph.spec, cmake, debian: use thrift 0.13+ from distro pkg

the change to build and ship libthift was added when we didn't have 0.13.0
version shipped via distro pkgs, now that centos 8 and F34 supports req.
version, we do not need to build and ship it with jaeger library.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #42342 from dsavineau/cephadm_template_jinja2_lint

mgr/cephadm/templates: add jinja2 lint

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

ceph.spec: enable WITH_JAEGER=ON if --with-jaeger passed

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake, ceph.spec, debian: use yaml-cpp >= 0.6

* since focal and centos both have yaml-cpp 0.6 available, which dropped
having boost as it's dependency, moving to 0.6 seems a good upgrade.

* cmake: delete Buildyaml, since distro suppilies v0.6 this is not needed

This fixes the build failure, as jaegertracing requires yaml-cpp v0.6+
```
Could NOT find yaml-cpp: Found unsuitable version "", but required is at
  least "0.5.1" (found yaml-cpp_LIBRARY-NOTFOUND)

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

vstart.sh: use quay to pull jaeger instead of docker

In servers like the ones available in sepia labs, users might hit rate
limiting for docker pull, it is better to use quay image to avoid this
issue.

https://blog.container-solutions.com/dealing-with-docker-hub-rate-limiting

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

doc/cephfs/journaler: document options using confval directive

better maintainablity this way. and drop unsupported options of

- journaler batch interval
- journaler batch max

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add "mypy" back to tox envlist of "qa""

This reverts commit 286e46578dcf35ab096dd242338f3751c248683c.

since 0017df2006bda64973799ab50735477674c3a5cd has been merged, let's
add mypy back.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/vstart_runner: add optional "sudo" param to _run_python()

to silence mypy warnings like:

tasks/vstart_runner.py:691: error: Definition of "_run_python" in base class "LocalCephFSMount" is incompatible with definition in base class "CephFSMount"
tasks/vstart_runner.py:705: error: Definition of "_run_python" in base class "LocalCephFSMount" is incompatible with definition in base class "CephFSMount"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42284 from orozery/remove-calc-target-vector-copies

osdc/Objecter: avoid vector copies in _calc_target

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42401 from tchaikov/wip-cmake-version

cmake: drop set(VERSION ...)

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #42408 from tchaikov/wip-mgr-always-on

mon/MgrMonitor: drop nautilus modules from always_on_modules

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42415 from sebastian-philipp/haproxy-user-root

cephadm: haproxy 2.4 defaults to a different container user.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #41907 from kamoltat/wip-ksirivad-progress-time-interval

pybind/mgr/progress: introduce 5 second sleep interval

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #42295 into master

* refs/pull/42295/head:
doc/cephfs: recover file system after recovering

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42402 from ljflores/perf-counter-docs

doc/dev/perf_counters: update docs to include more context about perf counter usage

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42333 from tchaikov/wip-doc-program

doc/man/8/ceph-volume: specify "program" for subcommands

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42196 from cbodley/wip-qa-rgw-rm-cephadm

qa/rgw: remove rgw_cephadm.yaml from rgw/singleton suite

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

doc/man/8/ceph-volume: specify "program" for subcommands

as per
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html

> Like py:currentmodule, this directive produces no output. Instead, it
> serves to notify Sphinx that all following option directives document
> options for the program called name.
> ...
> The program name may contain spaces (in case you want to document
> subcommands like svn add and svn commit separately).

and to avoid the warnings like:

doc/man/8/ceph-volume.rst:424: WARNING: Duplicate explicit target name:
"cmdoption-ceph-volume-h".

we should specify different "program" for different set of options.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42317 from cbodley/wip-39657

rgw multisite: metadata sync treats all errors as 'transient' for retry

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #41800 from BryceCao/wip-modfiy-XML-when-deleterole-take-error

rgw : modfiy error XML for deleterole

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41858 from BryceCao/wip-del-tenant-parse-in-createrole

rgw : add check for tenant provided in RGWCreateRole

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #42361 from cbodley/wip-49747

qa/rgw: add failing tempest test to blocklist

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #35153 from BryceCao/wip-swift-create-bucket-constraint

rgw : skip bucket name detection for system request

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35061 from joke-lee/rgw-multisite-forward-policy-accessblock

rgw: forward request in multisite for RGWDeleteBucketPolicy and RGWDeleteBucketPublicAccessBlock

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42273 from dabukalam/object-lock-retain-date-iso8601

rgw: Dump Object Lock Retain Date as ISO 8601

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #42425 from cbodley/wip-42788

test/rgw: fix use of poll() with timers in unittest_rgw_dmclock_scheduler

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42410 from ronen-fr/wip-ronenf-standalone-repair

qa/standalone: fixing the timings when waiting for deep-scrub to start

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #42430 from tchaikov/wip-cmake-qa-drop-mypy

cmake: drop "mypy" from tox envlist of "qa"

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42418 from tchaikov/wip-crimson-nbd

crimson/tools/store_nbd: pass app.alien() down to FSDriver

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42367 from tchaikov/wip-cmake-Og

cmake: add "-Og" to CMAKE_C_FLAGS_DEBUG

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #42427 from tchaikov/wip-seastore-metric-cleanup

crimson/os/seastore: trade a map with a plain array

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #42429 from neha-ojha/wip-51638-cleanup

qa/*/test_envlibrados_for_rocksdb.sh: remove OS specific configuration

Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: drop "mypy" from tox envlist of "qa"

this change partially reverts 81305b0da9a3e04633c022b12644e2975aa46a0d,
otherwise we have following errors:

tasks/vstart_runner.py:691: error: Definition of "_run_python" in base class "LocalCephFSMount" is incompatible with definition in base class "CephFSMount"
tasks/vstart_runner.py:705: error: Definition of "_run_python" in base class "LocalCephFSMount" is incompatible with definition in base class "CephFSMount"

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: trade a map with a plain array

simpler this way, as we don't need a std::map<> at all.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42261 from cyx1231st/wip-seastore-metrics-2

crimson/os/seastore: implement metrics at cache level

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42386 from liu-chunmei/seastore-metrics

crimson/seastore: add seastore operation latency metrics

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/rgw: use spawn library for test_rgw_dmclock_scheduler

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41929 from sebastian-philipp/fix-qa-tox

qa: Various make check fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42419 from dsavineau/fix_typo_radosgw_admin_max_entries

doc/man/8/radosgw-admin: fix typo on max entries

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42289 from mgfritch/cephadm-test-create-daemon-dirs-prom

cephadm: use pyfakefs during test_create_daemon_dirs_prometheus

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #42028 from mgfritch/cephadm-infer-mismatch-fsid

cephadm: infer fsid from ceph.conf

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41280 from sebastian-philipp/test_cephadm-stdin

qa/workunits/test_cephadm: Also test stdin

Reviewed-by: Michael Fritch <mfritch@suse.com>

doc/man/8/radosgw-admin: fix typo on max entries

This fixes a typo for the --max-entries option.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

crimson/tools/store_nbd: mark padding variable with [[maybe_unused]]

also silences -Wunused-variable warning from GCC

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd: pass app.alien() down to FSDriver

this was an oversight in 8fad523d41008727c9a952afcda15b77be50ba3c

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephadm: haproxy 2.4 defaults to a different container user.

Another alternative would be to investigage a different setup
leverageing `--sysctl net.ipv4.ip_unprivileged_port_start=0`,
but that would be a larger PR.

Fixes: https://tracker.ceph.com/issues/51355
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm/templates: add jinja2 lint

This adds a jinja2 lint environment in tox for testing the cephadm jinja2
templates.

This patch fixes some minor jinja2 syntax for ganesha and keepalived even if
the current templates work perfectly.

Tags should have one (and only one) space

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

cmake: use string(APPEND ...) to concat strings

less repeating this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add "-Og" to CMAKE_C_FLAGS_DEBUG

without specifying -Og, -O0 is used. as per
https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html

> -O0
>  Reduce compilation time and make debugging produce the expected
>  results. This is the default.

and

> -Og
>  Optimize debugging experience. -Og should be the optimization level of
>  choice for the standard edit-compile-debug cycle, offering a reasonable
>  level of optimization while maintaining fast compilation and a good
>  debugging experience. It is a better choice than -O0 for producing
>  debuggable code because some compiler passes that collect debug
>  information are disabled at -O0.

and Debug is the default built type if .git directory is found under
the root source directory, so by adding "-Og -g" to CMAKE_C_FLAGS_DEBUG,
developers can have better debugging experience when testing Ceph
built from a git repo. but the downside is that it might take longer
to build the tree.

this change should also enable us to link crimson on aarch64.

the same applies to Clang,

> -O0 Means “no optimization”: this level compiles the fastest and generates the most debuggable code.
> -O1 Somewhere between -O0 and -O2.
> -O2 Moderate level of optimization which enables most optimizations.
> -Og Like -O1. In future versions, this option might disable different optimizations in order to improve debuggability.

see
https://clang.llvm.org/docs/CommandGuide/clang.html#cmdoption-o0

see also https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101448

Fixes: https://tracker.ceph.com/issues/51441
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42413 from galsalomon66/fix_s3tests_yaml

RGW:revert force-branch to be ceph-master (s3-tests)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42304 from sebastian-philipp/doc-dameon-add-note

doc/cephadm: add notes to `orch daemon add`

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #42372 from sebastian-philipp/doc-cephadm-rgw-ssl

doc/cephadm: Add RGW ssl

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

revert force-branch to be ceph-master

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

crimson/os/seastore/cache: report dirty extents usage

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: report cached extents usage

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: measure read transaction efforts that are successful

In order to calculate the ratio of discarded vs successful effort from
read-only transactions.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: measure transactional efforts that are discarded or committed

The efforts of a transaction include the number and bytes of its read,
mutate, retire and fresh extents, and the bytes of delta generated.

This helps to understand the following aspects:
* The ratio of discarded efforts vs committed efforts;
* The average efforts of a transaction;
* The distribution of read/mutate/delta/retire/fresh efforts;
* The memory overhead and potential disk overhead of a transaction;
* How early a transaction invalidation happens;
* The average extent length;

It is possible to extend the effort metrics to be labeled by extent
types, in case we want to distinguish and profile the efforts at the
sub-component level.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: assert the committing delta is not empty

It makes no sense to commit an empty delta. It is mostly an issue that
user forget to generate delta during mutation, or there are futile
copy-on-write operations.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: cleanup to use Cache::invalidate(t, conflicting_extent)

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: count transactional cache access/hit

Labeled by source and extent-type.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42396 from ronen-fr/wip-ronenf-debug-docker-1

script: fix a bug in naming the source branch in ceph-debug-docker.sh

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42355 from yuvalif/wip-yuval-fix-51681

rgw/notifications: send correct size in case of delete marker creation

Merge pull request #42358 from yuvalif/wip-yuval-fix-51687

rgw/notification: send current time in complete multipart upload event

Merge pull request #42189 from yuvalif/wip-yuval-fix-51530

rgw/http/notifications: support content type in HTTP POST messages

crimson/seastore: add seastore operation latency metrics

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

README.md: recommend -Og instead of -O0

per https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html

> -Og
>  Optimize debugging experience. -Og should be the optimization level of
>  choice for the standard edit-compile-debug cycle, offering a reasonable
>  level of optimization while maintaining fast compilation and a good
>  debugging experience. It is a better choice than -O0 for producing
>  debuggable code because some compiler passes that collect debug
>  information are disabled at -O0.

so instead of using -O0, for better developer experience, -Og is
recommended.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/standalone: fixing the timings when waiting for deep-scrub to start

initiate_and_fetch_state() initiates a scrub, then polls the published
PG state looking for 'scrubbing'. Calling flush_pg_stats() as part of
the polling process might cause the scrub and the following recovery to
be missed altogether.

Note: this polling mechanism is definitely not robust. Will be
redesigned in the future.

Fixes: https://tracker.ceph.com/issues/51581
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

mon/MgrMonitor: drop nautilus modules from always_on_modules

the list of always_on_modules for the older versions is only for the
upgrade use case, where the monitor is still the old release, so
we have to keep an entry for the old release in the always_on_modules
map.

but since we don't upgrade from a release older than 2 releases, there
is no need to keep an entry for nautilus.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/mgr/administrator: document mgr_disabled_modules

this option prevent mgr from loading specified modules. which could
be handly when debugging issues with always_on_modules.

and document mgr_standby_modules as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42388 from tchaikov/wip-crimson-alien

crimson/osd: use app.alien() to initialize AlienStore::alien

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42392 from tchaikov/wip-test-test_ipaddr

test/test_ipaddr: always set ifa_flags

Reviewed-by: Neha Ojha <nojha@redhat.com>

test/rgw: fix use of poll() with timers in unittest_rgw_dmclock_scheduler

the AsyncScheduler uses an asio timer to dispatch work to its executor
with an optional delay. when no delay is requested, it waits on the
timer with an expiration time in the past (crimson::dmclock::TimeZero)

tests are failing here because poll() is returning without executing the
handlers of those expired timers

asio implements these timers with timerfd and epoll. debugging with
strace, i see that these timers armed with timerfd_settime() are not
always immediately ready according to epoll_wait():

  eventfd2(0, EFD_CLOEXEC|EFD_NONBLOCK)   = 3
  epoll_create1(EPOLL_CLOEXEC)            = 4
  timerfd_create(CLOCK_MONOTONIC, TFD_CLOEXEC) = 5
  epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLERR|EPOLLET, data={u32=14164052, u64=14164052}}) = 0
  epoll_ctl(4, EPOLL_CTL_ADD, 5, {events=EPOLLIN|EPOLLERR, data={u32=14164064, u64=14164064}}) = 0
  timerfd_settime(5, TFD_TIMER_ABSTIME, {it_interval={tv_sec=0, tv_nsec=0}, it_value={tv_sec=0, tv_nsec=1}}, {it_interval={tv_sec=0, tv_nsec=0}, it_value={tv_sec=0, tv_nsec=0}}) = 0
  epoll_wait(4, [{events=EPOLLIN, data={u32=14164052, u64=14164052}}], 128, 0) = 1
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [{events=EPOLLIN, data={u32=14164064, u64=14164064}}], 128, 0) = 1

in this example, it took 6 calls to context.poll() before it was ready
to execute the timer's handler

to work around this, replace calls to context.poll() with calls to
context.run_for() with a very short duration

Fixes: https://tracker.ceph.com/issues/42788
Signed-off-by: Casey Bodley <cbodley@redhat.com>

debian/control: update yaml-cpp version to 0.6

This fixes the build failure, as jaegertracing requires yaml-cpp v0.6+
```
Could NOT find yaml-cpp: Found unsuitable version "", but required is at
  least "0.5.1" (found yaml-cpp_LIBRARY-NOTFOUND)
```
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

debian, install-deps.sh: use Build-Profiles to optionally build debian libjaeger
library

* use pkg.ceph.jaeger for debian optional pkgs
remove mangling needing install-deps, instead use, buildProfileSpec feature
introduced for debian.  https://wiki.debian.org/BuildProfileSpec * check and

* set extraopts in debian/rules using pkg.ceph.jaeger
see: https://github.com/ceph/ceph/pull/38783#discussion_r662995612

* cleanup libjaeger.install mangling from CMakeLists

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

doc/dev/perf_counters: update docs to include more context about perf counter usage

The Perf Counters docs, although informative, are lacking for users and developers who are wondering what they can do with their perf counter data. I wrote an extra paragraph here that outlines some ways in which the counters can be used, including diagnosing problems in a cluster and identifying workload patterns.

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #42188 from votdev/issue_51408_motd

mgr/dashboard: Add configurable MOTD or wall notification

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

cmake: drop set(VERSION ...)

project(...). this is the authentic place to set version number.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42102 from Matan-B/lua-request-userId

rgw: Added request field to lua scripting