cmake: amend linking issues for jaeger

apart from using distro yaml-cpp and thrift pkgs, this adds:

* custom target jaeger_base
* target jaeger-base which encapsulates all jaeger libs

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake: delete Buildyaml, since distro suppilies v0.6 this is not needed

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake: change debian DESTDIR for installing jaeger deps

debian uses debian/tmp as destination dir for installing build files,
but since we are using common path($build_dir/external) available both
for rpm and debian based dependency installation, it becomes far more
complicated to maintain include/link path for these external projects.

elaborating on it:
path we are configuring for both rpm and debian installing including,
and linking of external librarires:
/build/ceph-17.0.0-5779-g928f9e55/obj-x86_64-linux-gnu/external/
debian appends DESTDIR to this path, and hence our predefined target
artificats cannot find correct path for external libs, I tried adding
ENV${DESTDIR} so that it could include correct external lib install
path, but it still cannot find them:
eg:
https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=focal,DIST=focal,MACHINE_SIZE=gigantic/56123//consoleFull
Since the install path is in our build environment for these librarires,
skipping DESTDIR looks to me hacky fix, but does the job.
with empty destdir:
https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=focal,DIST=focal,MACHINE_SIZE=gigantic/56090//consoleFull

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake: tweak jaeger build requirements

what changes:

* target name Jaeger >> jaegertracing : since jaegertracing is the
original target that jaeger submodule uses in it's cmake, cmake build
complained if named otherwise

* remove redundant call build_jaeger

* Boost is a dependency for jaeger, to use the right version, we pass
ceph build boost path to cmake jaeger build step

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

ceph.spec: enable WITH_JAEGER=ON if --with-jaeger passed

needed for cmake optional build options during jenkins shaman builds

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake: use $dependencies to track tragets that'd be build as external project

jaeger depends on thrift, opentracing, boost, yaml-cpp some of them are
available as distro pkgs, if not available we track and build these
using ${dependencies} variable.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake: move ExternalProjectHelper & IncludeJaeger to BuildJaeger

set_library_properties_for_external_project assists with setting right target
properties for all jaeger dependencies.
IncludeJaeger would take care of linking and creating these targets
having them spread out when they are highly coupled seems not optimal.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

ceph.spec: use yaml-cpp >= 0.6

since focal and centos both have yaml-cpp 0.6 available, which dropped
having boost as it's dependency, moving to 0.6 seems a good upgrade.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

ceph.spec: use thrift from rpm distro pkgs

the change to build and ship libthift was added when we didn't have 0.13.0
version shipped via distro pkgs, now that centos 8 and F34 supports req.
version, we do not need to build and ship it with jaeger library.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake, debian: do not build thrift as ext project, adds yaml-cpp by default

we need specific versions of thrift(0.13.0+) and yaml-cpp(0.6+) for jaeger to
work properly, earlier they were not supported by distros, but since we moved
to use centos 8 and focal, that seems not be the case.

In this case we shall use the find_package cmake method for thrift and yaml-cpp.
this change updates:

* moves yaml-cpp find/build to src/CMakeLists(as seastar might also need it)
* adds findthrift.cmake and builds thrift if v0.13.0 not found >> adds the same
to debian/libjaeger as pkg files needed

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake: cleanup debian/libjaeger.install mangling

we will now be using BuildProfileSpec based optional pkg building,
removing comment from .install file is no longer needed
If pkg.ceph.jaeger is enabled debian/control shall work to install jaeger and
it's dependencies

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

vstart.sh: use quay to pull jaeger instead of docker

In servers like the ones available in sepia labs, users might hit rate limiting for docker pull, it is better to use quay image to avoid this issue.

https://blog.container-solutions.com/dealing-with-docker-hub-rate-limiting

Signed-off-by: Deepika <dupadhya@redhat.com>

debian/control: update yaml-cpp version to 0.6

This fixes the build failure, as jaegertracing requires yaml-cpp v0.6+
```
Could NOT find yaml-cpp: Found unsuitable version "", but required is at
  least "0.5.1" (found yaml-cpp_LIBRARY-NOTFOUND)
```
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

debian, install-deps.sh: use Build-Profiles to optionally build debian libjaeger
library

* use pkg.ceph.jaeger for debian optional pkgs
remove mangling needing install-deps, instead use, buildProfileSpec feature
introduced for debian.  https://wiki.debian.org/BuildProfileSpec * check and

* set extraopts in debian/rules using pkg.ceph.jaeger
see: https://github.com/ceph/ceph/pull/38783#discussion_r662995612

* cleanup libjaeger.install mangling from CMakeLists

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake: update build<lib>.cmake for supporting ninja

* affects cmake builds for: yaml-cpp, thrift, jaeger, opentracing external
projects

changes done:
* artificat BUILD_BYPRODUCT which tells ninja which library will be generated
after the build(needed for dependent build libs)
* use cmake_command if `make` not found so that we use default generator, cmake
will identify and use ninja directly in this case.
* minor reorder of boost path in jaeger build external projet

fixes: https://tracker.ceph.com/issues/51029

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

install-deps.sh: add missing with_jaeger option for ceph.spec mangling

Signed-off-by: Deepika <dupadhya@redhat.com>

Merge pull request #42144 from cyx1231st/wip-fix-seastore-cache2

crimson/os/seastore/cache: fix retiring mutation-pending extents

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@360.cn>

crimson/os/seastore/cache: fix retiring mutation-pending extents

Mark the retiring mutation-pending extent as INVALID, and add it's
prior-instance to retired-set in order to populate transaction
invalidation correctly.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42143 from athanatos/sjust/wip-cm-omap-int-future

crimson/os/seastore: convert omap and collection_manager to interruptible future

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42138 from rzarzynski/wip-crimson-alienstore-indent

crimson/os: fix indentation in alien_store.cc.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42128 from tchaikov/wip-crimson-asock-cleanup

crimson/asock: cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os/seastore/omap_manager: convert to use interruptible future

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/collection_manager: convert to use interruptible_future

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore: clean up transaction_manager based tests

Permits using either vanilla TransactionManager and
InterruptedTransactionManager, updates users to use submit_transaction
helpers.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #41980 from yuvalif/wip-yuval-fix-51320

rgw/notification: support version-id for all event types

Merge pull request #42127 from ivancich/wip-little-rgw-cleanup

rgw: a small set of clean-ups

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #42012 from yuvalif/wip-yuval-fix-50039

rgw/notifications: support v4 auth for topics and notifications

rgw/notification: support version-id for all event types

including: object copy, multipart upload, delete marker on versioned bucket

Fixes: https://tracker.ceph.com/issues/51320
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #42134 from zdover23/wip-doc-cephadm-operations-2021-07-01

doc/cephadm: improve "Ceph Daemon Logs" (1 of x)

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42023 from TRYTOBE8TME/wip-rgw-notification-tests

rgw: Modification in the usage of force-branch

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #41945 from yuvalif/wip-yuval-fix-51261

rgw/notifications: support metadata filter in CompleteMultipartUpload and Copy events

Merge pull request #41953 from SMIL-Infra/fix-dashboard-osd-count

mgr/dashboard: fix OSD out count

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #42116 from votdev/issue_51443_migrate_userdb

mgr/dashboard: User database migration has been cut out

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

crimson/os: fix indentation in alien_store.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #42107 into master

* refs/pull/42107/head:
mgr/ActivePyModules: streamline mgr_ips a bit
mgr/ActivePyModules: don't take unnecessary lock
pybind/mgr/mgr_module: use get("mgr_ips") for standby get_mgr_ip()
mgr: add get() for standby modules

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

doc/cephadm: improve "Ceph Daemon Logs" (1 of x)

This PR turned out to be a 3-in-1:

(1) improves syntax and formatting of "Logging to stdout"
(2) improves syntax and formatting of "Logging to files"
(3) replaces all carets with tildes in 3rd-level section
    headers in operations.rst (./build-doc was crying
    about inconsistency when I fed it tildes, but tildes
    and not carets are the RST standard according to
    https://docutils.sourceforge.io/ \
    docs/user/rst/quickstart.html#sections
    so the carets had to go.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #42130 from tchaikov/wip-doc-install

doc/install/get-packages: s/el7/el8/ and cleanups

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42108 from zdover23/wip-doc-upgrading-ceph-cancelling-the-upgrade-2021-06-30

doc/cephadm: improve "Canceling an Upgrade"

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc/install/get-packages: do not mention additional 3rd party packages

since we are able to build and run on ubuntu focal without extra
dependencies of 3rd party packages, there is no need to mention them.
actually all of the removed packages are available in ubuntu focal.
some of them have different package names though.

the same applies to el8.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/install/get-packages: s/el7/el8/

since we don't build for el7 anymore. replace all occurrences of el7 with
el8.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: Modification in the usage of force-branch

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

crimson/admin: let AdminSocket::register_command() return void

instead of returning a seastar::future<>, it can just return void,
as it does not involve asynchronous operations.

also drop servers_tbl_rwlock, as we don't register hooks after OSD
starts, there is no need to protect hooks with a rw lock.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: remove unused variable

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42080 from cyx1231st/wip-seastore-fix-cache

crimson/os/seastore/cache: misc fixes and cleanup

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@360.cn>

crimson/os/seastore/cache: rename try_construct_record to prepare_record

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: wait io when read from Cache::extents

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: add the missing extent invalid check

see 2bc257beb2ef1253e38c254745f42b3ac5db52de

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: add missing add_to_read_set()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: cleanup get_extent_if_cached()

And remove the query_cache_for_extent(t).

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: fix get_extent_by_type(t) to use Transaction::get_extent()

The original query_cache_for_extent() doesn't call wait_io() if the
extent is pending. And the following get_extent_by_type() already covers
the lookup to Cache::extents and also implements wait_io() correctly.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: add absent extent to read-set before retire

So that transaction can be invalidated if the retiring extent has
conflict.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: remove retired extents from cache

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: fix print_detail() of RetiredExtentPlaceholder

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: cleanup try_construct_record() optional return

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: check invalid transaction only

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: retire PRESENT extents and add to retired-set

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: invalidate transaction referencing retired extents

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42103 from athanatos/sjust/wip-seastore-collection-ordering

crimson/seastore: enforce per-collection transaction ordering

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xuxuehan@360.cn>

Merge pull request #41932 from tchaikov/wip-mgr-mock-osdmap

mgr,pybind/mgr: add OSDMap.build_simple() method

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #41958 from ceph/wip-yuriw-notes-14.2.22_2

doc: 14.2.22 Release Notes

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/releases/nautilus,octopus: add a note for msgr2 interoperability fix

octopus 15.2.13 already shipped, let's add it retroactively.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc/releases/nautilus: add notable changes for RADOS

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc: 14.2.22 Release Notes

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #42118 from sandrobonazzola/patch-3

doc/rbd/iscsi-initiator-linux: set "product" in multipath.conf

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/rbd/iscsi-initiator-linux: set "product" in multipath.conf

Adding missing parameter in multipath configuration.  See
https://bugzilla.redhat.com/show_bug.cgi?id=1769135.

Signed-off-by: Sandro Bonazzola <sbonazzo@redhat.com>

rgw: a small set of clean-ups

Small items noticed along the way and addressed collectively in this
small "omnibus" PR.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #42121 from ceph/wip-md

ceph-release-notes: Support for markdown for new ceph.io site

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-release-notes: Support for markdown for new ceph.io site

Signed-off-by: David Galloway <dgallowa@redhat.com>

mgr/ActivePyModules: streamline mgr_ips a bit

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/ActivePyModules: don't take unnecessary lock

This isn't needed.

Signed-off-by: Sage Weil <sage@newdream.net>

pybind/mgr/mgr_module: use get("mgr_ips") for standby get_mgr_ip()

This is necessary to allow the standby mgr to bind to the right IP.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr: add get() for standby modules

Fixes: https://tracker.ceph.com/issues/51446
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #42026 from rhcs-dashboard/bucket-name-regex-fix

mgr/dashboard: Fix bucket name input allowing space in the value

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41353 from a16bitsysop/mgr-node_modules

pybind/mgr/CMakeLists.txt: exclude files not used at runtime

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: User database migration has been cut out

This PR will revert changes done by https://tracker.ceph.com/issues/49645 to auto-migrate user database v1 to v2.

Fixes: https://tracker.ceph.com/issues/51443
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #41954 from pkalever/notrim

rbd-nbd: support notrim option with map command

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42089 from tchaikov/wip-xattr-cleanup

osd, crimson/osd: do_cmp_xattr() related cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #42054 from tchaikov/wip-os-xattrs

os: use transparent comparator in ObjectStore::getattrs()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #42043 from tchaikov/wip-ceph-argparse-ann

pybind/ceph_argparse: do not set self.typeargs in ctor of CephArgtype

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42087 from tchaikov/wip-common-buffers

common/buffers: check _num directly in list::c_str()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42084 from tchaikov/wip-49638

qa: run e2e test on centos only

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #42047 from tchaikov/wip-autoscaler-ann

pybind/mgr/pg_autoscaler: add typing annotations and cleanups

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42082 from fengchunsong/master

common: Use double instead of long double to improve performance

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37866 from pritha-srivastava/wip-sts-47809

rgw/sts: fix for copy object operation using sts

qa/workunits/rbd-nbd: add notrim test

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

mgr/dashboard: Fix bucket name input allows space in the value

Fixes: https://tracker.ceph.com/issues/51368
Signed-off-by: Nizamudeen A <nia@redhat.com>

pybind/mgr/pg_autoscaler: add typing annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/pg_autoscaler: do not index pool_root with pool_name

pool_root is indexed by pool_id, and we never index it with pool_name.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/pg_autoscaler: do not create a tuple long_desc

long_desc is supposed to be a str, not a tuple of str.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/pg_autoscaler: extract CrushSubtreeResourceStatus out

as it also serves as part of interface of get_subtree_resource_status(),
not only its internals. to ease adding the type annotations, this class
is promoted out of the class.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/pg_autoscaler: define commands using CLICommand

simpler this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/pg_autoscaler: define options using Option

more consistent and less error-prune this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/cephadm: improve "Canceling an Upgrade"

This PR improves the section "Canceling an Upgrade"
in the "Upgrading Ceph" chapter of the cephadm
documentation.

I removed an extraneous prompt and rewrote a sentence
so that it was congruent with other sentences in similar
places elsewhere in the documentation.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41919 from agayev/zoned-more-cleaning-support

os/bluestore: More support for cleaning zones.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #42100 from rzarzynski/wip-crimson-assert_moveable

crimson: introduce assert_moveable().

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind/ceph_argparse: add more type annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41976 from tchaikov/wip-crimson-alienstore-lockless

crimson/os: use lockfree queue for sharded queue

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #42101 into master

* refs/pull/42101/head:
cephsqlite: add comment on atexit

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: add collection ordering

Adds a mutex to SeastoreCollection which ensures that transactions are
ordered on a per-collection basis.  Future optimizations could enable
the transaction construction phase to be pipelined for a single collection,
but would require correctly handling the case where a more recently
submitted transaction encounters a conflict.

Fixes: https://tracker.ceph.com/issues/51358
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/seastore: preserve Transaction::handle on reset for writes

We're going to need to preserve the pipeline phase locking on conflict.

Signed-off-by: Samuel Just <sjust@redhat.com>