win32*.sh: fetch WNBD dependency

Due to a rebase mistake, we're no longer fetching WNBD, which
is required by rbd-wnbd.

This commit will take care of cloning WNBD and generating an
import library.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

win32*.sh: bump win32 boost dependency

Ceph now requires boost>=1.72. We're going to update the Windows
build script accordingly.

There's been one Boost regression for which we have to cherry-pick
a patch that hasn't been released yet.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

cmake: skip osd, Lua and neorados on Windows

Those libraries are currently unsuported on Windows, so we're
going to update the cmake file to skip them.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

win32*.sh: improve lib handling

When building dependencies, we assume that libs will be placed in the
"lib" dir. Still, some distros might use "lib64" by default.

For this reason, we'll explicitly pass the expected lib path.

At the same time, we're dropping an unnecessary lib copy.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

win32*.sh: update openssl location

We'll fetch the openssl source code from the github repository,
since the download links from the official website tend to expire
quite often.

At the same time, we'll drop the version from the directory name.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

win32*.sh: cleanup dependency build dir when targetting Windows

We're using a separate dir when building Ceph dependencies for
Windows. The build script isn't entirely idempotent ATM, so the
easiest thing to do is to just cleanup this dir when rebuilding
dependencies.

If *all* dependencies have been successfully built, this step is
skipped by default.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

win32*.sh: Strip binaries individually

Instead of copying over a rather large directory, strip symbols off
of files individually.

Signed-off-by: Mike Latimer <mlatimer@suse.com>

win32*.sh: Cleanup zip archive creation

Use a temporary directory and symlinks to create the final zip archive
within a single 'ceph' parent directory. Also include the win32 README,
and report the name of the resulting zip file.

Signed-off-by: Mike Latimer <mlatimer@suse.com>
Signed-off-by: Alessandro Pilotti <apilotti@cloudbasesolutions.com>
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

cmake,win32*.sh: Add OS=(ubuntu|suse) flag, for distro settings

Distributions such as SUSE and Ubuntu differ significantly in their mingw
environments. This adds an OS=(ubuntu|suse) flag, which can be used to
specify which environment is being used for the build. Unless set explicitly,
the scripts will try to automatically detect it.

Depending on the OS selected, various mingw configuration options (binaries,
library paths, etc.) as well as required packages are determined.

Due to these options being configured at runtime, corresponding cmake
files are generated on the fly.

Signed-off-by: Mike Latimer <mlatimer@suse.com>
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Signed-off-by: Alessandro Pilotti <apilotti@cloudbasesolutions.com>

win32*.sh: Use nproc for CPU count when targeting win32

When building in chroot jail, /proc/cpuinfo may not be available. Switch to
nproc which should be widely available, and does not rely on /proc/cpuinfo.

Signed-off-by: Mike Latimer <mlatimer@suse.com>

test: Include compat.h to find uid_t/gid_t/gidlist when cross compiling

This change is necessary to prevent the following type of messages
(and compile failures) when cross-compiling:

/src/include/cephfs/libcephfs.h:148:29: error: 'uid_t' was not declared in this scope; did you mean 'pid_t'?
  148 | UserPerm *ceph_userperm_new(uid_t uid, gid_t gid, int ngids, gid_t *gidlist);
      |                             ^~~~~
      |                             pid_t

/home/abuild/rpmbuild/BUILD/ceph-15.1.0-1521-gcdf35413a0/src/include/cephfs/libcephfs.h:148:40: error: 'gid_t' was not declared in this scope; did you mean 'pid_t'?
  148 | UserPerm *ceph_userperm_new(uid_t uid, gid_t gid, int ngids, gid_t *gidlist);
      |                                        ^~~~~
      |                                        pid_t

Signed-off-by: Mike Latimer <mlatimer@suse.com>

win32*.sh: improve win32 dependency check

At the moment, we assume that the dependencies have been successfully
built if the "build.deps" directory exists.

For convenience reasons, this change will use a file instead,
signaling that we've finished building the dependencies. This will
allow the build process to be resumed when interrupted.

Signed-off-by: Mike Latimer <mlatimer@suse.com>
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

win32*.sh: isolate network operations when building win32 dependencies

We'll move network operations (git/wget/curl) under a conditional
check of the source directory.

This will ensure that the script is idempotent and that network
operations can be avoided, using pre-existing sources if available.

Signed-off-by: Mike Latimer <mlatimer@suse.com>
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

cmake: drop Python dependency when targeting Windows

The Python bindings aren't supported on Windows for the time being,
for which reason we're dropping the Python dependency.

When cross-compiling, Cmake sometimes complains about not finding
the Python files instead of just using the host files.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

win32*.sh: cleanup Windows build script

Since the switch to Python 3, a few Cmake flags are no longer required.
While at it, we're dropping a few other unused Cmake flags.

We're going to pass the right target os flag when building lz4
and make sure that it's included in our zip archive.

win32_deps_build.sh needs a small change, fixing a sed command to become
idempotent.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Signed-off-by: Mike Latimer <mlatimer@suse.com>

win32*.sh: fix patch command

There's a leftover "cat" command used when patching Boost files,
which will wait for stdin. This commit fixes this issue.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

win32*.sh: update Windows build scripts and readme

We're going to allow building a zip containing stripped binaries
and skipping the ceph tests. We'll use a separate option rather
than changing the build type so that may generate binaries
containing debug symbols as well as stripped binaries in one shot.

While at it, we're updating the Windows readme.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Signed-off-by: Mike Latimer <mlatimer@suse.com>

Merge pull request #37971 from lixiaoy1/rwl_plugin

librbd/cache: make image persistent writeback cache as plugin

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38484 from rhcs-dashboard/update-ssl-error-1

mgr/dashboard: Updating the inbuilt ssl providers error

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38459 from ronen-fr/wip-ronenf-lua-include

rgw/lua: fixing incompatible declarations of CephContext

Merge pull request #37519 from pcuzner/docs-cephadm-config-drift

doc/dev/cephadm: high level design for a compliance check feature

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38498 from bk201/wip-48506

mgr/dashboard: remove pyOpenSSL version pinning

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38415 from insujang/master

msg/async: refine worker creation in NetworkStack

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38457 from tchaikov/wip-mon-scrub-interval

mon: use TYPE_SEC for mon_scrub_interval

Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mgr/dashboard: remove pyOpenSSL version pinning

Tests require a newer version of pyOpenSSL.

Fixes: https://tracker.ceph.com/issues/48506
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #38497 from tchaikov/wip-seastar

seastar: update seastar submodule

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38491 from agayev/fix-hmsmr-compilation

os/bluestore: Fix HMSMRDevice.cc compilation.

Reviewed-by: Kefu Chai <kchai@redhat.com>

seastar: update seastar submodule

to pick up recent fixes in file io

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38437 from ifed01/wip-ifed-alloc-replay-dump

os/bluestore: add more means to troubleshoot allocator(s)

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #38454 from tchaikov/wip-compressor-cleanup

compressor: put Compressor into TOPNSPC namespace and use string_view

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38458 from tchaikov/wip-librados-cct

librados: move definition of cct_deleter to .cc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38418 from anthonyeleven/anthonyeleven/clarify-op-priorities

doc: clarify osd recovery op priority and fix a couple of typos

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

os/bluestore: Fix HMSMRDevice.cc compilation.

Signed-off-by: Abutalib Aghayev <agayev@psu.edu>

Merge PR #38419 into master

* refs/pull/38419/head:
vstart_runner: print result line unless opt_rotate_log is set

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Adding the alert bad certificate error to the ssl providers error

upstream tracked in https://github.com/cherrypy/cheroot/pull/348
Fixes: https://tracker.ceph.com/issues/48490
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Updating the inbuilt ssl providers error

Fixes: https://tracker.ceph.com/issues/48490
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #37956 from dorindabassey/example-notific

examples/boto3/README: examples notification

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #38374 from rzarzynski/wip-crimson-backfill-remove

crimson: add support for backfill, part 2 -- MOSDPGBackfillRemove

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38392 from rzarzynski/wip-crimson-backfill-virtual_pgfacade

crimson: use explicit, abstract interfaces for backfill's facades.

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38208 from sebastian-philipp/mgr-cephadm-type-annotations-mandatory

mgr/cephadm: make type annotations mandatory for some modules

Reviewed-by: Michael Fritch <mfritch@suse.com>

crimson: use explicit, abstract interfaces for backfill's facades.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #37130 from pcuzner/cephadm-exporter

cephadm:Add a daemon mode for cephadm to provide a metadata endpoint

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #38483 from tchaikov/wip-crimson-cleanup

crimson: cleanups

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/cephadm: make annotations for schedule.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: make annotations for inventory.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: make annotations for serve.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: make annotations for upgrade.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38106 from varshar16/wip-fix-rook-orch-ps-img-id-name

mgr/rook: Fix container id and image id in 'orch ps'

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #38346 from varshar16/wip-remove-container-id-field-orch-ps

mgr/orchestrator: Remove 'CONTAINER ID' col from orch ps if unknown

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #37952 from mgfritch/cephadm-json-decode

mgr/cephadm: handle JSONDecodeError

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38032 from mgfritch/cephadm-last-local-image-name

cephadm: reference the last local image by digest

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38064 from mgfritch/cephadm-rm-host-key-err

mgr/cephadm: verify the `host` key exists during refresh

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

crimson/test: make the backfill testing quieter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/test: make use of maybe_flush() in the backfill testing.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for MOSDPGBackfillRemove at primary.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for MOSDPGBackfillRemove at replicas.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38130 from sebastian-philipp/cephadm-upgrade-fail-daemon

mgr/cephadm: upgrade: fail gracefully, if daemon redeploy fails

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38317 from sebastian-philipp/cephadm-reduce-log-dg-noise

mgr/cephadm: OSD service: reduce log noise

Reviewed-by: Michael Fritch <mfritch@suse.com>

crimson/osd: return something in PG::do_delete_work()

this silences the warning of

crimson/osd/pg.cc:316:1: warning: no return statement in function returning non-void [-Wreturn-type]
  316 | }
      | ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: use a constexpr function for sanity test

since match_stage_t is an alias of uint_8. it is an unsigned type, so an
instance of this type should be always greater or equal to 0.

this silences warning like:

stage_types.h:37:50: warning: comparison is always true due to limited range of data type [-Wtype-limits]
   37 |     static_assert(STAGE >= STAGE_BOTTOM && STAGE <= STAGE_TOP);
      |                                            ~~~~~~^~~~~~~~~~~~

also, GCC complains if it is able to assure that the compare always
returns true. so a function helps to silence it.

this warning is meaningless per-se, as we are using static_assert() to
perform sanity test at *compile-time*.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd/cache: move image writeback cache to plugin

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #38460 from ronen-fr/wip-ronenf-lua-move

rgw: removing a pessimizing move

doc: clarify osd recovery op priority and fix a couple of typos

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

librbd/plugin: enable hook points for exclusive lock

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #36986 from cyx1231st/wip-seastar-onode-tree-upstream

crimson/seastore: add initial onode staged-fltree

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/test: fix test/tool make target names

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/staged-onode-tree: use LT, EQ, GT for comparison results

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: add and fix comments

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #38356 from icepic/patch-2

doc/radosgw: Fix misspelled word in index.rst

Reviewed-by: Zac Dover <zac.dover@gmail.com>

rgw: removing a pessimizing move

"warning: moving a temporary object prevents copy elision"

Co-authored-by: Kefu Chai <kefu@redhat.com>
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #38324 from tchaikov/wip-mon-cleanups

mon: drop QuorumService and cleanups.

Reviewed-by: Joao Eduardo Luis <joao@suse.com>

mon/ConfigKeyService: drop ConfigKeyService::dispatch()

this method is basically an alias of
ConfigKeyService::service_dispatch(), so get rid of it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: drop ConfigKeyService::get_name()

the only caller of this function is prefix(), so hardcode the service
name there instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: remove unused bits

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: store mon and paxos using reference

before this change, we always have a valid pointer as member variable.
so, a reference serve this purpose better.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: drop QuorumService

so far we only implemented ConfigKeyService, so move QuorumService into
the its only child class -- ConfigKeyService.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: move private member variable declaration to the end of class

to follow https://google.github.io/styleguide/cppguide.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: store a reference as member variable

before this change, we always have a valid pointer as member variable.
so, a reference serve this purpose better.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: use unique_ptr<> to manage lifecycle of Monitor::paxos

it is less error-prone, and has better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw/lua: fixing incompatible declarations of CephContext

Existing forward declaration breaks OSD compilation (clang)

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

librados: move definition of cct_deleter to .cc

so cct->get() and cct->put() are visually closer. this improves the
readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: use TYPE_SEC for mon_scrub_interval

* common/legacy_config_opts.h: drop mon_scrub_interval from legacy
  settings. as this option is not in critical path. so we can afford
  the overhead of look its value up in the map.
* common/options.cc: change mon_scrub_interval's type from TYPE_INT
  to TYPE_SECS, so we can use human friendly representations like
  "3 days" when specifying its value
* mon/Monitor.cc: adapt to the changes above.
* doc/rados: update the doc for "mon scrub interval" accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/Timer: add SafeTimer::add_event_after() which accepts timespan

easier to use when we are moving from the number of seconds to a
C++ timespan.

Signed-off-by: Kefu Chai <kchai@redhat.com>

compressor: pass string_view instead of string

this allows us to find by string without creating a std::string
instance, even if this string is not nul terminated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

compressor: put Compressor into TOPNSPC namespace

we want to add the support for on-the-wire compression to msgr v2, and
this feature will be shared by classic osd and crimson. but
Compressor.cc is also used by bluestore which is linked against by
crimson also. Compressor depends on CephContext which has two different
implementation in two namespaces for classic osd and crison.

so to avoid violating ODR, we should also put Compressor into two
different namespaces so it can be shared by alienstore and crimson in
the same executable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38442 from badone/wip-bump-fmt-package-min-version

cmake: require fmt 6.0.0 and up

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38389 from trociny/wip-48412

librbd: fix regression in object map diff request

Reviewed-by: Jason Dillaman <dillaman@hotmail.com>

Merge pull request #35821 from Aran85/fix-ec-read

osd/ECBackend: optimize remaining read as readop contain multiple objects

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #37543 from ianwatsonrh/master

docs: With the addition of STS x-amz-security-token is now supported

Merge pull request #38401 from a16bitsysop/alpine-tests

test/lazy-omap-stats: Fix compilation on alpine linux

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38314 from badone/wip-monitor-reset-abort-if-is_shutdown

mon: paxos: Delete logger in destructor

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38410 from badone/wip-monitor-logger-deelete-in-destructor

mon: Delete logger in destructor

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37314 from ifed01/wip-ifed-faster-rm-p1

osd: optimize PG removal (part1)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #38438 from ifed01/wip-ifed-fix-avl

os/bluestore: fix inappropriate ENOSPC from avl/hybrid allocator

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38436 from ktdreyer/prometheus-err

mgr/prometheus: don't store exception as e

Reviewed-by: Boris Ranto <branto@redhat.com>

cmake: require fmt 6.0.0 and up

The cmake search for a local package has lagged in regard to our usage
of it.

Fixes: https://tracker.ceph.com/issues/48453
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #38400 from tchaikov/wip-pybind-use-non-deprecated-functions

pybind/rados: use rados_set_pool_full_try instead of the deprecated one

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36384 from linuxbox2/wip-doc-ippolicy

rgw:doc: document support for (Not)IpAddress in S3 policy

os/bluestore: bring back probing available space for main dev allocator.

This feature was mistakenly removed while making a single allocator
for main device (https://github.com/ceph/ceph/pull/30838).

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #37546 from myoungwon/wip-revamp-tier-flush

osd: refactoring tier_flush()

Reviewed-by: Samuel Just <sjust@redhat.com>