crimson: shutdown messengers before deallocate OSD

Stop dispatching events from messenger when OSD is about to stop and
deallocate.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: implement shard-local messenger internally

Adopt FixedCPUServerSocket and don't move sockets across cores after
connected/accepted.

Implement the messenger to be managed in one CPU only, since we have
encapsulated the seastar listen-on-all requirement inside
FixedCPUServerSocket, and the requirements of a cross-core messenger has
not been defined yet.

The messenger interfaces can also be simplified, but will be in another
patch.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: test messenger only on designated core

Messenger/Connection are not implemented properly to be a cross-core
service, and the related use cases are not defined yet.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: configure seastar to accept on a fixed core

Adopt FixedCPUServerSocket and don't move sockets after
connected/accepted.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: implement FixedCPUServerSocket

Adopt the policy load_balancing_algorithm::fixed to instruct seastar
(posix-stack) to allocate connected sockets on the designated core, so
we are not able to move them later in our application, which is not
supported by seastar and result in undefined behaviors.

Seastar requires server_socket to accept/listen on all available cores.
We encapsulate the related implementations in the new
FixedCPUServerSocket, so we can have a simpler shard-local crimson
messenger.

Even though FixedCPUServerSocket is a sharded service, only the service
located on the shard where the socket is originally created will be
serving incoming connections. it's not allowed to perform i/o with that
connected socket on other cores.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #33066 from gregsfortytwo/wip-specfile

rpm: fix up a specfile syntax error

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #30033 from theanalyst/rgw/public-buckets

rgw: add PublicAccessBlock set of APIs on buckets

Reviewed-By: Casey Bodley <cbodley@redhat.com>

rpm: fix up a specfile syntax error

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #33049 from yuvalif/wip-yuval-fix-43768

qa/rgw/pubsub: fix tests to sync from master

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: add Pending Release Notes entry on public access config

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #32119 from joke-lee/rgw-sts-crash-duration-invalid

rgw: fix rgw crash when duration is invalid in sts request

Merge pull request #31987 from linuxbox2/rgw-putacls-no-body

rgw: s3: don't require a body in S3 put-object-acl

Merge pull request #30684 from theanalyst/rgw/qa/rgw-admin-user-stats

qa: radosgw_admin: validate a simple user stats output

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge PR #33048 into master

* refs/pull/33048/head:
cephadm: fix error handling in `command_check_host()`
cephadm: increase default retry_max value

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32995 into master

* refs/pull/32995/head:
cephadm: add group 'disk' to privileged container
cephadm: adopt: disable ceph-volume unit on host
cephadm: refactor privileged arg handling
cephadm: chown block symlink targets
cephadm-adoption-corpus: add stud-mon.tgz
cephadm: adopt: rename leveldb *.ldb -> *.sst
cephadm: adopt: by default, pull first
cephadm: adopt: chown data content

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33020 into master

* refs/pull/33020/head:
osdc/Objecter: inline pool full check
osdc/Objecter: remove duplicated pause check code
osdc/Objecter: only pause if respects_full()
osdc/Objecter: move respects_full() to op_target_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32831 into master

* refs/pull/32831/head:
common, include: drop the copy{_in} from bufferlist entirely.
os/bluestore: switch copy_in() users to bufferlist::iterator.
osdc: switch users of bufferlist::copy{_in} to iterators.
osd: switch users of bufferlist::copy{_in} to iterators.
rgw: switch copy{_in} users to bufferlist::iterator.
ec: switch users of bufferlist::copy{_in} to iterators.
cls/queue: switch users of bufferlist::copy{_in} to iterators.
client: switch users of bufferlist::copy{_in} to iterators.
*: switch trivial users of bufferlist::copy{_in} to iterators.
test/bl: switch copy{_in} users to bufferlist::iterator.
common, include: kill the bl::last_p member.
common: encode for std::list<T> doesn't use bl::copy_in() anymore.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32531 from zdover23/wip-doc-landing-page-update

doc: Added the crisp getting started guide to index.rst

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cephadm: add group 'disk' to privileged container

This lets the osd read block devs that are group rw disk even after they
drop root privs.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32977 into master

* refs/pull/32977/head:
qa/workunits/cephadm/test_cephadm.sh: add missing monitoring tests
cephadm: simplify Monitoring.components structure
cephadm: add proper tox type for monitoring components

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge PR #33012 into master

* refs/pull/33012/head:
mgr/cephadm: prefix daemon ids with hostname
cephadm: bootstrap: name mgr with $hostname.$random

Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: adopt: disable ceph-volume unit on host

This might be a simple or lvm unit.  Disable it so that the host doesn't
try to start this OSD after a reboot.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: refactor privileged arg handling

Pass a bool if we want a privileged container instead of explicitly
passing --privileged.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fix error handling in `command_check_host()`

`find_program()` raises `ValueError` when the executable hasn't been
found. It means we need to catch `ValueError` exception in
`command_check_host()` and raise `Error` instead of `RuntimeError` since
only `Error` is caught at the end.

Typical failure:

```
INFO:cephadm:/usr/bin/ceph:stderr Error ENOENT: New host mon1 failed check: ['INFO:cephadm:podman|docker (/bin/podman) is present', 'INFO:cephadm:systemctl is present', 'Traceback (most recent call last):', '  File "<stdin>", line 2820, in <module>', '  File "<stdin>", line 2434, in command_check_host', '  File "<stdin>", line 796, in find_program', 'ValueError: lvcreate not found']
```

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

rgw: move public access conf to perm_state_base

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

test: rgw_iam_policy update tests with new Get Actions

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: iam_policy: add all the actions to actpairs map

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: public access: drop unused function

Also cleanup the comment to mention why we've deviated from the spec here

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: rename ACL & policy IsPublic to is_public

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: PublicAccessConfiguration -> PublicAccessBlockConfiguration

also drop iam namespace

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_op: get_public_access_from_attr indent fixes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_rest_s3: use formatter->dump_bool

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

public_access: reuse formatter->dump_bool

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: Fix IgnorePublicACLs for bucket ACLs

Currently Bucket ACLs with IgnorePublicACLs were broken this should fix that

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

drop redundant bucket policy status in rgw_common

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: implement IgnorePublicACLs

This allows for ignoring bucket/object acls that are configured to be public

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: move PublicAccessConfiguration to req_state

This allows for evaluation of more complex use cases where IgnorePublicACLs and
the like are set which need to be evaluated for GET/HEAD requests as well

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: op: Get Policy Status checks if there is a policy first

before evaluating its public nature

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_op: reuse function to get public access conf

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: honor PublicAccessBlockConfiguration for put object

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: block public access for Policies

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: enforce BlockPublicPolicy on put bucket policy

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: public_access: rename the getters to reflect what the conf actually does

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_op: use ldpp_dout version wherever logs are required

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: move IsPublic to RGWAccessControlPolicy class

This helps reusing when evaluating for PutACLs and Put Policy

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: implement get/put/delete public access block for buckets

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

add RGWPublicAccess for configuring public access settings for a resource

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: both princ and nonprinc will not coexist in a single statement

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: initial implementation of a public policy tester

doesn't cover all the cases involving a nonprinc user yet

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: s3: implement GetBucketPolicyStatus API

This API returns whether the Bucket Policies/ACLs are public. There are a couple
of caveats:
- AWS currently returns PolicyNotFound error in case a bucket policy doesn't
exist, though a non existant bucket policy would mean the default ACLs apply
where the bucket is private, so error return here seems like an error
- the API spec mentions TRUE and FALSE as the response IsPublic element value,
however in practice both boto/aws clients and AWS S3 return/expect a lowercase
response.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
Conflicts:
src/rgw/rgw_rest_s3.h
merge conflict after zipper rework, dropped a spurious newline in rgw_rest_s3.h
after get_obj_op decl.
src/rgw/rgw_common.h
src/rgw/rgw_rest_s3.cc
src/rgw/rgw_rest_s3.h:
merge conflict after bucket replication merge, trivial conflicts

rgw: acl: drop unused function & make get_group_perm const

Drop the unused RGWAccessControlPolicy::get_group_perm, make the ACL
get_group_perm as a const member function

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

cephadm: increase default retry_max value

When playing with cephadm, at multiple times, I've reached the max
number of attempt in `is_available()`

Increasing the `retry_max` helps to avoid failure like following:

```
INFO:cephadm:mgr not available, waiting (1/5)...
INFO:cephadm:mgr not available, waiting (2/5)...
INFO:cephadm:mgr not available, waiting (3/5)...
INFO:cephadm:mgr not available, waiting (4/5)...
INFO:cephadm:mgr not available, waiting (5/5)...
ERROR: mgr not available after 5 tries

```

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #32155 from yuvalif/wip-yuval-multidelete-notifications

rgw/pubsub: send notifications from multi-delete op

mgr/cephadm: prefix daemon ids with hostname

This is friendlier to a human operator since they can immediately see
where an instance is located, as with the legacy scheme, while still
keeping the unique random suffix.  Use a . to separate so that we can
set per-host options.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32859 from liewegas/wip-cephadm-services

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33030 from smithfarm/wip-luminous-12.2.13-release-notes

doc: release notes for Luminous v12.2.13

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #33031 from smithfarm/wip-nautilus-14.2.7-release-notes

doc: release notes for Nautilus 14.2.7

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #33038 from tchaikov/wip-librados-reinterpret-cast

librados: prefer reinterpret_cast over c-style cast

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #32933 from DimStar77/obs-mini

ceph.spec.in: Use pkgconfig() style BuildRequires for udev/libudev-devel

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32553 from tspmelo/wip-ts-strict

mgr/dashboard: Enable compiler options used by Angular --strict flag

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

librados: prefer reinterpret_cast over c-style cast

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33016 from tchaikov/wip-do-cmake

do_cmake.sh: bail out if something goes wrong

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/workunits/cephadm/test_cephadm.sh: add missing monitoring tests

add tests for node-exporter, prometheus, and grafana

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: simplify Monitoring.components structure

by removing the nested image/metadata dict

Signed-off-by: Michael Fritch <mfritch@suse.com>

More commits to the landing page project.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

rgw/pubsub: fix tests to sync from master

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

cephadm: bootstrap: name mgr with $hostname.$random

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32570 into master

* refs/pull/32570/head:
cephfs-shell: Add tests for setxattr, getxattr and listxattr
cephfs-shell: Add listxattr command
cephfs-shell: Add getxattr command
cephfs-shell: Add setxattr command
doc: Update about extended attributes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31633 into master

* refs/pull/31633/head:
cephfs-shell: Instead of assert use stat for tests in rmdir
cephfs-shell: Add function for common rmdir test code
cephfs-shell: Add rmdir test for non empty directory
cephfs-shell: Add rmdir -p test for non empty directory
cephfs-shell: Add rmdir -p test for non existing dir
cephfs-shell: Add rmdir -p test to delete all dirs in given path
cephfs-shell: Add rmdir -p test for root directory with empty directories
cephfs-shell: Add rmdir test for valid file
cephfs-shell: Add rmdir test for invalid directory
cephfs-shell: Add rmdir test for valid directory
cephfs-shell: Fix rmdir '-p' issues

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32986 into master

* refs/pull/32986/head:
qa/tasks/ceph_manager: fix movement of cot exports with cephadm

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc: release notes for Nautilus 14.2.7

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: release notes for Luminous v12.2.13

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #32030 from vshankar/wip-mgr-volumes-clone

mgr/volumes: clone from snapshot

Merge PR #33014 into master

* refs/pull/33014/head:
mgr/dashboard: pass a list of drive_group to create_osds

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32973 into master

* refs/pull/32973/head:
mgr/insights: make 'insights prune-health-history 0' zap current state

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32987 into master

* refs/pull/32987/head:
qa/tasks/ceph_manager: make fix_pgp_num behave when no pool is found

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32989 into master

* refs/pull/32989/head:
qa/tasks/ceph_manager: add --log-early to raw_cluster_cmd

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32830 into master

* refs/pull/32830/head:
global: ensure CEPH_ARGS is decoded before early arg processing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32806 into master

* refs/pull/32806/head:
common/bl: fix accessibility of bptr's _off and _len fields.
common/bl: drop get_raw() from the public buffer::ptr interface.
common: drop sharing of buffer::raw outside bufferlist.

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph_manager: fix movement of cot exports with cephadm

I think this will finally work...

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: chown block symlink targets

Again, these need to match the container's uid/gid, not the host's.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm-adoption-corpus: add stud-mon.tgz

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: adopt: rename leveldb *.ldb -> *.sst

We may be adopting from ubuntu, which means we need to rename these files.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: adopt: by default, pull first

This just bit me: I had a months-old container revision locally and ended
up accidentally starting up ancient monitor code.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: adopt: chown data content

The ceph uid/gid vary between distros.  Chown everything to match the
target distro.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: inline pool full check

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: remove duplicated pause check code

_calc_target() already pauses and unpauses based on the
target_should_be_paused() helper, and we just called _calc_target at the
top of this method.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: only pause if respects_full()

If the FULL_TRY or _FORCE flag is set on an op, then we should not pause
it due to a full flag.

Fixes: https://tracker.ceph.com/issues/43813
Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: move respects_full() to op_target_t

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33017 from linuxbox2/wip-rgw-beastc

rgw: improve beast

Merge pull request #33010 from tchaikov/wip-cmd-getval

src/: drop cct from cmd_getval()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

do_cmake.sh: bail out if something goes wrong

so we can be aware if some updated submodule reference is missing in our
repo

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: pass a list of drive_group to create_osds

as orchestrator backends expect a list of drive_groups.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Unable to remove an iSCSI gateway that is alrea… (#32984)

mgr/dashboard: Unable to remove an iSCSI gateway that is already in use

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add Modified filter to Configuration table (#32923)

mgr/dashboard: Add Modified filter to Configuration table

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

rgw: improve beast

Avoid leaking connections that had partially-consumed
client data on unexpected disconnect.

Resolves CVE-2020-1700 (moderate impact flaw).

Fixes: https://tracker.ceph.com/issues/42531
Signed-off-by: Or Friedmann <ofriedma@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

cephadm: add proper tox type for monitoring components

instead of `type: ignore`

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge branch 'octopus'

mgr/dashboard: Enable noImplicitAny TS compiler option

Fixes: https://tracker.ceph.com/issues/43525
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Enable noImplicitReturns TS compiler option

Fixes: https://tracker.ceph.com/issues/43525
Signed-off-by: Tiago Melo <tmelo@suse.com>