cephadm: only SIGKILL once during timeout

no need to continually SIGKILL a process that is already exiting

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add profile log line to Popen wrappers

add debug log line to profile the Popen wrapper read loop
exec runtime, exitcode, stop, fds to read, etc.

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: increase is_available timeout 30s -> 60s

bootstrap fails because `ceph -s` might take longer
than 30 sec to return on resource limited hardware

Fixes: https://tracker.ceph.com/issues/45961
Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35099 from sebastian-philipp/doc-orch-update-specs

doc/orch: How to update an existing specification

Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #35506 from rzarzynski/wip-crimson-thread-comment-fix

crimson/thread: fix comment phrasing in ThreadPool.h.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34492 from xxhdx1985126/wip-crimson-recovery-block-io-pr

crimson/net: fix system shutdown problems

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/thread: fix comment phrasing in ThreadPool.h.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #35423 from scoopex/master

tools/histogram_dump: improve io-latency and -size statistics

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35187 from CodingSpiderFox/patch-3

doc: Introduce the concept of a pool before talking about it.

Merge pull request #32705 from petrutlucian94/windows.3

include,common: Windows support [part 3]

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35422 from adamemerson/wip-45878

osdc: Fix crash on OSD failure

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33720 from yunfeiguan/wip-44409

log: fix timestap precision of log can't set to millisecond.

Reviewed-by: Adam Emersen <aemerson@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #34667 from aclamk/wip-rocksdb-reshard

Resharding tool for sharded rocksdb

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35186 from liewegas/wip-cas

cls_cas: misc cleanup, and dynamic chunk_refs_t

Reviewed-by: Samuel Just <sjust@redhat.com>

tools/histogram_dump: improve io-latency and -size statistics

The script is part of the ceph distribution but not
part of the official tooling. Nevertheless it was useful to understand
the usage scenarios of my osds.

- add colors to the output to ease interpretation
- add possibility to specify one ore more sockets
- aggregate statistics over multiple sockets
- prevent stacktrace if tool is interrupted by SIGINT
- add possibility to specify the number of data collection
  seconds between loops
- raise loop_seconds default

Signed-off-by: Marc Schoechlin <ms-github@256bit.org>

doc/orch: How to update an existing specification

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34633 from sebastian-philipp/cephadm-total-scheduler

mgr/cephadm: make scheduling make sense

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35460 from agayev/fix-cmake

blk: Add missing PUBLIC keyword to target_link_libraries command.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35095 from tchaikov/wip-kefu-onode-v1

crimson/os: add onode node implementation

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os: add OnodeBlock

used for representing the an node of onode tree

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add crimson::gtest target

instead re-compiling gtest_seastar.cc, better off creating a static
library and link against it instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: add node implementation for onode tree

* node_t and friends are representations of in-mem and on-disk onode
  block/page/node. it might have different names in different contexts.
  but normally, they are the same thing and are loaded as a logical
  cached extent.
* delta_t holds the mutations against a given node onode. they are
  applied to memory once they are committed to disk as part of the
  log record.

regarding to the name of "simple-fltree", it is a short for simple
Flexible Layout Fixed Tree, which was designed by Samuel Just. My
implementation is a naive implementation of this design. But it
tries to be extensive and should be able to adapt to different policies
for rebalancing nodes when overflow/underflow. But there could be
other variant of flexible layout tree implementations in future,
so i added "simple" prefix.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: extract onode.h out

so the concrete onode manager implementation can include it without
including the onode_manager.h interface as a whole

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: add ObjectStore and OnodeManager stubs

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #33878 from theanalyst/rgw/cleanup/user-policy-append

rgw: append user policies to existing policies

Reviewed-by: Pritha Srivastava <prsivas@redhat.com>
Reviewed-by: Adam Emersen <aemerson@redhat.com>

Merge pull request #33521 from linuxbox2/wip-getset-ver-nosuch

rgw: fail when get/set-bucket-versioning attempted on a non-existent …

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #33511 from theanalyst/rgw/append-gc-fixes

rgw: append obj: prevent tail from being GC'ed

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33492 from prasakri/multisite-sync-check

RGW:Multisite: Verify if the synced object is identical to source

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #35470 into master

* refs/pull/35470/head:
mds: fix compile error with CephContext declaration

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35344 into master

* refs/pull/35344/head:
vstart: Support deployment of ganesha daemon by cephadm with NFS option.
mgr/test_orchestrator: List nfs daemon

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #35298 from tspmelo/wip-ng-progressbar

mgr/dashboard: Use ng-bootstrap for Progressbar

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #33083 from mkogan1/wip-rgw-add-beast-access-logs

rgw: add access log to the beast frontend

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

src: Rename "zns" directory to "zoned".

"zoned" is a more appropriate name for both HDDs and SSDs that support the zone
interface.  ZNS is an NVMe standard for zoned SSDs only.  Zoned HDDs, on the
other hand, use a SATA standard called ZBC/ZAC.

Signed-off-by: Abutalib Aghayev <agayev@cs.cmu.edu>

src: Add missing PRIVATE keyword to target_link_libraries command.

This fixes the following cmake error that happens if you have libzbc installed:

CMake Error at src/blk/CMakeLists.txt:40 (target_link_libraries):
  The keyword signature for target_link_libraries has already been used with
  the target "blk".  All uses of target_link_libraries with a target must be
  either all-keyword or all-plain.

  The uses of the keyword signature are here:

   * src/blk/CMakeLists.txt:32 (target_link_libraries)

Signed-off-by: Abutalib Aghayev <agayev@cs.cmu.edu>

Merge pull request #35379 from votdev/issue_45856_show_host_labels

mgr/dashboard: Show labels in hosts page

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mds: fix compile error with CephContext declaration

SimilarTo: https://github.com/ceph/ceph/pull/35209
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rgw: add access log to the beast frontend

Add to the Beast frontend an access log line similar to CivetWeb.
attempting to adhere as much as possible to the Apache Combined Log
Format.

Fixes: https://tracker.ceph.com/issues/45920
rgw: use beast message for access log
(cherry picked from commit 44ec38933cbe6cc864c56332cc99502d86568fdc)

Co-authored-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #35381 from tspmelo/wip-fix-cephfs-directories

mgr/dashboard: Fix CephFS's Directories not showing

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35373 from bk201/wip-45870

mgr/dashboard: remove space after service name in the Hosts List table

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35457 from tchaikov/wip-rpm-unitdir

test/crimson: avoid using variadic future

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: avoid using variadic future

see also 3cee25ba41069e9e9b4855f99be20910759440ca

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35405 from ivancich/wip-fix-truncated-bucket-listing

rgw: bucket list/stats truncates for user w/ >1000 buckets

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #33762 from clwluvw/wildcard-subuser

rgw: Add support wildcard subuser for bucket policy

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #32393 from clwluvw/opa-access-key

rgw: Add subuser to OPA request

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35203 from linuxbox2/wip-lc-perfctr

rgwlc: add lifecycle perf counters

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35073 from cbodley/wip-45382

rgw: add quota enforcement to CopyObj

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Or Friedmann <ofriedma@redhat.com>

Merge pull request #34041 from mdw-at-linuxbox/wip-rgw-swstat-loops-2

rgw: fix loop problem with swift stat on account.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: convert rgw_user_max_buckets to new config options

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: clean up code around buckets listing

This commit adds some const correctness, avoids some string/structure
copying, and reformats some code.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: bucket list/stats truncates for user w/ >1000 buckets

The is_truncated flag was not being handled properly and therefore
limited the output on these radosgw-admin commands to 1000 buckets.

Fixes: https://tracker.ceph.com/issues/45630
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #35380 from sebastian-philipp/cephadm-logs-daemon-id

cephadm: More robust way to deduce the systemd unit

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35411 from sebastian-philipp/adopt-9cd9ad020d93b0b420924fec55da307aff8bd422

qa/cephadm: update adopt corpus

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35372 from mgfritch/orch-service-spec-mypy

python-common: add ServiceSpec type checking

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

osdc: Fix crash on OSD failure

OSDSession::lock should not use lockdep, as we violate its rules in a
place that's safe. Revert it to std::mutex.

Thanks to Or Friedmann <ofriedma@redhat.com> for finding the problem.

Fixes: https://tracker.ceph.com/issues/45878
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #34275 from pritha-srivastava/wip-rgw-sts-oidc-provider

rgw: adds code for creating and managing oidc provider entities in rgw and for offline validation of OpenID Connect Access and ID Token.

rgw: adding code for policy evaluation for ops
like getbucketversioning, putbucketversioning etc

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: Correcting pool name for oidc provider for older
versions of osd.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: fix for make failure in test folder.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adds code for role session to be used an ARN.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adding new iam ops for iam oidc provider
to correct unittest faliure.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: modifying documentation to include explanation of
open id connect provider related REST APIs and removing
references to token introspection for validating incoming
web token.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adds a commom method to remove prefix(www, http and https)
from a url.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adds code to check for client id and cert thumbprint
while validating incoming web token in AssumeRoleWithWebIdentity.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: Added code for offline OpenId Connect/ OAuth 2.0 tokens
if the token is a JWT.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: Adding jwt-cpp header only library for JWT validation.
https://github.com/Thalhammer/jwt-cpp

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adds REST APIs for OpenID connect providers manipulation.

This commit implements:
1. CreateOpenIDConnectProvider
2. DeleteOpenIDConnectProvider
3. ListOpenIDConnectProviders
4. GetOpenIDConnectProviders

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #35340 from mgfritch/cephadm-srv-id-host

mgr/orch: allow '.' char in hostname

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35413 from sebastian-philipp/upgrade-empty-realm

doc/cephadm: Adopt single-site RGW

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Use ng-bootstrap for Progressbar

Fixes: https://tracker.ceph.com/issues/45752
Signed-off-by: Tiago Melo <tmelo@suse.com>

cephadm: More robust way to deduce the systemd unit

Generate a better error message when using wrong args.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35365 from sebastian-philipp/doc-orch-unmanaged

doc/mgr: add docs for unmanaged=True

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35311 from tspmelo/wip-ng-tooltip

mgr/dashboard: Use ng-bootstrap for Tooltip

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

vstart: Support deployment of ganesha daemon by cephadm with NFS option.

NFS Ganesha cluster can be created in two ways with vstart
cluster:

1) With test_orchestrator
NFS=1 ../src/vstart.sh
This type of deployment can have multiple ganesha daemons on random port.

2) With Cephadm
NFS=1 ../src/vstart.sh --cephadm
It can deploy only single ganesha daemon with vstart on default ganesha port.

Both can create multiple cephfs exports.

Fixes: https://tracker.ceph.com/issues/45830
Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/test_orchestrator: List nfs daemon

This patch adds dummy data for nfs ganesha daemon and adds nfs to list of
supported service type.

Signed-off-by: Varsha Rao <varao@redhat.com>

doc/cephadm: Adopt single-site RGW

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

crimson/osd: fix memory leaks

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/mon: return -EBUSY, instead of asserting, when MonClient::active_con is absent

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/net: wait for wandering connections to close when shutting down

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/monc: close pending connections when stopping

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: make osd shutdown graceful

The shutdown operation OSD is as follows:
1. call each components' stop in the inner-out order, interrupting all
continuations by setting the corresponding promises with the exception
system_shutdown_exception;
2. wait for all inflight continuations to complete;
3. reclaim resources

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

qa/cephadm: update adopt corpus

to 9cd9ad020d93b0b420924fec55da307aff8bd422

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Use ng-bootstrap for Tooltip

Fixes: https://tracker.ceph.com/issues/45754
Signed-off-by: Tiago Melo <tmelo@suse.com>

doc/mgr: add docs for unmanaged=True

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35297 from tspmelo/wip-alert

mgr/dashboard: Use ng-bootstrap for Alert

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35375 from sebastian-philipp/doc-cephadm-volker-error

doc/cephadm: Add alternative error for missing CIDR network

Reviewed-by: Volker Theile <vtheile@suse.com>

crimson/recovery: fix obc check issue

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

mgr/dashboard: remove space after service name in the Hosts List table

Fixes: https://tracker.ceph.com/issues/45870
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/orch: use keyword rather than positional arg

use keyword `service_id` from the orch cli

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #35263 into master

* refs/pull/35263/head:
test_cephfs_shell: use StringIO instead of BytesIO

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

test_cephfs_shell: use StringIO instead of BytesIO

Code in qa/ uses both StringIO and BytesIO. Let's use StringIO
exclusively (unless necessary) for uniformity. The reason for using
StringIO over BytesIO is that tests mostly need stdout as string than
as bytes and StringIO is used more frequently used in qa/ code at this
point.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #35271 from bk201/refactor-template-cephadm

mgr/cephadm: use jinja2 to render templates

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33646 from ofriedma/wip-mkogan-ofriedma-add-cache-api

rgw: Adding data cache and CDN capabilities

Reviewed-by: Mark Kogan <mkogan@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35018 from sebastian-philipp/cephadm-rm-container

cephadm: Manually remove containers

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

common: stub certain modules and functions on Windows

We're going to stub certain modules and functions for Windows builds.
Some of them will be implemented in subsequent commits, some are platform
specific and will be skipped.

Modules:
* module.c
* subprocess.cc
* blkdev.cc
* dns_resolve.cc
* numa.cc
* syslog.h
* statvfs.h
* libblk
* neorados

Functions:
x signal functions
x fork
x chown
* socketpair_cloexec
* dump_open_fds
* run_cmd
* is_symlink
* CInode::d_type
* nbd commands

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Signed-off-by: Alin Gabriel Serdean <aserdean@cloudbasesolutions.com>

include/buffer: Extend win32 platform check

<sys/mman.h> is unavailable on Windows. There's a check
for __CYGWIN__ that will have to be extended to look for
_WIN32.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

include/compat: Add win32 compatibility layer

Windows provides socket and other related structures that are mostly
following the Posix standards, but using different headers.

Rather than adding lots of platform checks and require future commits
to do so as well, we're adding headers with the same names, which
in turn include the required Windows headers.

In a few cases, some functions declared by unistd.h are actually
defined by different Windows headers, so we'll need additional includes.

One thing to note here is that boost requires us to include aio.hpp
before the Windows socket headers.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

include/compat: Add missing win32 type and constant definitions

The following types and constants are not defined on Windows, for
which reason we're going to include them in compat.h.

* uid_t, gid_t
* sigset_t
* HOST_NAME_MAX
* O_CLOEXEC, which we're going to define as a no-op. By default,
  handles are not inherited by subprocesses.
* SHUT_* flags
* loff_t
* __STRING
* EBADE, ENODATA, ESHUTDOWN, ESTALE, EREMOTEIO
* suseconds_t
* SIGKILL, SIGINT - we're mostly going to stub signal handling
on Windows

Also, we'll have to make sure that pipe_cloexec uses C style linking,
which would otherwise be affected as the include order changes.

timeval.tv_sec is defined as long instead of time_t on Windows[1], which
we'll have to take into account.

[1] https://docs.microsoft.com/en-us/windows/win32/api/winsock/ns-winsock-timeval

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

mgr/dashboard: Use ng-bootstrap for Alert

Fixes: https://tracker.ceph.com/issues/45751
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35176 from aclamk/wip-prio-cache-name

common: PriorityCache's perf counters can have custom name

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #35299 from tspmelo/wip-ng-popover

mgr/dashboard: Use ng-bootstrap for Popover

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>