crimson/os/FuturizedCollection: use thread_safe_counter

This should work for now, long term we probably want to get
rid of FuturizedCollection entirely from the interface.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/heartbeat: drop ownership of messenger instances

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/pg_shard_manager: add shard_id asserts

Signed-off-by: Samuel Just <sjust@redhat.com>

src/vstart.sh: add vstart --crimson-smp option

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/net: Message::conn needs to be a foreign_ptr

There are two main consequences of this:
1. Messages can't be default copy constructed in crimson.  MMonCommand
   seems to be the only user, and we simply add a copy constructor that
   duplicates data portions of the message.
2. We can't casually copy-construct the conn into other structures.
   The main user here is watch/notify.  We use copy() explicitely
   to populate the object_context structures and avoid passing
   ConnectionFRef by value.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd_operations/client_request: remove OSD reference

Capture ShardServices reference for use in put_historic directly.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: move send_incremental_map to OSDSingletonState

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/net/Connection: let send and keepalive be called from foreign cores

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/shard_services: check core on each PerShardState method

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd_operations/client_request: fix whitespace on do_process

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: update_heartbeat_peers can iterate pgs syncronously

OSDSingletonState has a local inventory of all pgids.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd_operations: use foreign_ptr for conn

Signed-off-by: Samuel Just <sjust@redhat.com>

vstart: echo osd arguments

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd_operations/peering_event.cc: whitespace fix

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: remove compound_peering_request

The state shared between sub events creates problems for multicore.  The
only user is MOSDPGCreate2, so the optimization isn't really worth
salvaging.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/operation: release pipeline stages on the core they are on

Otherwise, tasks waiting on the stage will wake up on the wrong core.
Later, we may choose to statically enable this behavior only for stages
that can actually span cores, but this is ok for now.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: move perf counter ownership to OSDSingletonState

Later, we'll want to create a properly sharded implentation that avoids
barriers.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: shard PerShardState across cores

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: move hb_stamps to PerShardState

We may later need to syncronize this across cores, perhaps during tick.
This should work for now, however.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: replace global tid with unique core-local tids

We don't really want a global counter here if we can avoid it.  Instead,
assign tids with core-local counters prefixed with the core id.  We
continue to ensure that tids are unique within an osd boot, but lose
the property that sucessive tids on different cores are ordered.  I
don't see anything relying on that property, however, so this should be
fine.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: remove explicit pg counter

The main user of num_pgs was get_target_pg_log_entries(), but we
don't really want to be accessing a global counter there if we
can avoid it.  Instead, compute a core-local target and use the
core-local pg count.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: clean up PerShardState/OSDSingletonState ownership/access

- move heartbeat messenger ownership to OSD
- move OSDSingletonState/ShardServices initialization to
  PGShardManager::start() method
- add accessor OSD::get_shard_services(), remove OSD::shard_services
  reference
- accessor/forwarding cleanups within PGShardManager

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: move store responsiblity to PerShardState

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os: consolidate context handling in FuturizedStore

Context handling is pretty uniform accross all implementations,
may as well do it in the same place.

ShardedStoreProxy would need to handle it otherwise, since
callbacks need to be handled on the core do_transaction is
invoked on.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/futurized_store: create ShardedStoreProxy

For now, FuturizedStore implementations assume that methods are invoked
on core 0.  Later, we'll adapt each implementation to intelligently
support invocation on any pg core.  Until then, this wrapper converts
the existing implementations to a safe, if not particuarly performant,
proxy behavior.

AlienStore should be safe as is.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: adjust get_pg_stats() and for_each_pg() to return futures

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: refactor pg management for multicore

OSDSingletonState will now only be responsible for the spg_t->core
mapping for pgs, the individual PerShardState's will hold local
PGMap's.  PG management operations are now proxied from PGShardManager
to PerShardState.  Subsequent patches will shard PerShardState.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: modify start_pg_operation to return op_id

The op will actually be run generally on a different core, so we don't
want to be returning a reference to it.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/pg_map: add PGShardMapping

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: rework OSDMap handling for multicore

OSDMaps can occupy significant space in memory.  Duplicating OSDMaps
across cores would multiply that memory usage as well as complicate
the internal structure sharing we do when applying OSDMap incrementals.

Because PeeringState and other interfaces expect efficient copying of
OSDMapRef objects we don't want to use foreign_ptr directly.

Instead, maintain a single cache and distribute local_shared_foreign_ptrs
to other cores.  ShardServices becomes the only OSDMapService.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common: introduce local_shared_foreign_ptr

Introduces a foreign_ptr wrapper to allow core-local refcounting.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/operation: generalized for multiple reactors

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: make mkfs static

Can't access pg_shard_manager state here without invoking
start().  Instead, let's just be explicit about meta_coll
etc.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/main: need to actually invoke ConfigProxy::start()

sharded_proxy()::start() is a sharded<> method.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/main: don't shard OSD

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: rename CoreState to OSDSingletonState

CoreState was a bad name -- "core" is going to be used
to refer to seastar reactors/cores as well.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/shard_services: formatting fixes

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common: introduce smp_helpers.h, crimson::submit_to

Introduce crimson::submit_to wrapper to transparently deal with
vanilla and errorated futures, adds helpers to errorator.h.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../client_request: use intrusive_ptr for instance_handle

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #48121 from rzarzynski/wip-crimson-pgadvmap-critical-from

crimson/osd: initialize PGAdvanceMap::from inside critical section

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #48131 from adamemerson/wip-no-fno-new-ttp-matching

build: Remove -fno-new-ttp-matchingg flag

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48172 from mchangir/doc-add-note-about-cephfs-mirroring-paths

doc/cephfs: add note about cephfs-mirroring paths

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc/cephfs: add note about cephfs-mirroring paths

Signed-off-by: Milind Changire <mchangir@redhat.com>

Merge pull request #48149 from batrick/i57586

tools: allow spaces in dentries for first-damage.sh

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #48057 from athanatos/sjust/wip-io-hang

crimson/osd/client_request: use fresh tracking_events/handle instances on requeue

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd/client_request: use fresh tracking_events/handle instances on requeue

See instance_handle_t explanation in client_request.h

Fixes: https://tracker.ceph.com/issues/57494
Fixes: https://tracker.ceph.com/issues/57495
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #48059 from athanatos/sjust/wip-57506

crimson: fix for pgs stuck in +wait along with some debug improvements

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #48091 from rzarzynski/wip-crimson-pgadvmap-complete-in-order

crimson/osd: PGAdvanceMap ops complete in order

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #48134 from rhcs-dashboard/silence-fix

mgr/dashboard: use service call instead of form component

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

tools: allow spaces in dentries for first-damage.sh

Fixes: https://tracker.ceph.com/issues/57586
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: test first damage with dentry containing spaces

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #45369 from jyha200/wip-samplededup

ceph_dedup_tool: adding sample-dedup

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #41855 from p-se/compile-cephadm-1

cephadm: compile cephadm

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #47873 from pdvian/wip-log-last

mon/LogMonitor: Fix log last

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #47852 from Matan-B/wip-matanb-radosmodel

test/osd/RadosModel: Improve error debugging

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #47841 from Matan-B/wip-matanb-rados-striper

SimpleRADOSStriper: Avoid moving bufferlists by using deque in read()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #47774 from vumrao/wip-vumrao-57271

blk/KernelDevice: Modify the rotational and discard check log message

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #48156 from cbodley/wip-rgw-unused-throttles

rgw: remove unused AioThrottles in rgw_op.cc

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #46321 from jmolmo/issue_55605

mgr/rook: fix error when trying to get the list of nfs services

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #46961 from tcoldrick-bb/balancer-weekday

mgr/balancer: Disallow 7 as {begin,end}_weekday and 2400 as {begin,end}_time

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #46091 from pdvian/wip-bluestore-error

os/bluestore: Log error to stderr if fail to decode label

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #43791 from rzarzynski/wip-net-perfctrs-for-encrypted-traffic

auth, msg/async: add perf counters for encrypted traffic

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #48166 from rhcs-dashboard/fix-empty-pid-path-case

exporter: don't skip loop if pid path is empty

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

exporter: don't skip loop if pid path is empty

Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Fixes issue when pid file config comes empty from config dump which prevents to add metrics. Also get process metrics only if
pid_path isn't empty.

Merge pull request #47986 from lmgdlmgd/patch-3

osd: set per_pool_stats true when OSD has no PG

Reviewed-by: Igor Fedotov <ifedotov@croit.io>

Merge pull request #48139 from ktdreyer/win-simplify-openssl

win32: skip installing openssl man pages

mgr/dashboard: use service call instead of form component

For creating the silence from the notification sidebar, instead of using
the silence form which will require initializing the whole component on
the landing page, we can just call the prometheus service and pass on
the required data to the service call. This will fix showing the
`Prometheus not configured` error everytime we visit the landing page when
the prometheus is not configured

Fixes: https://tracker.ceph.com/issues/57576
Signed-off-by: Nizamudeen A <nia@redhat.com>

ceph_dedup_tool: fix threshold comparison

If threadhold count is 2, existing code always returns true
even though reference count is 1 actually  because it additionally
adds one via ++.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

ceph_dedup_tool: make SampleDedupWorkerThread::chunk_size const

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_dedup_tool: remove unused SampleDedupWorkerThread::duplicable_chunks

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_dedup_tool: minor code simplifications

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_dedup_tool: remove unimplemented mark_non_dedup method signature

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_dedup_tool: fix log lines to use std::endl, update object skip message

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_dedup_tool: remove unused SampleDedupWorkerThread::do_object_dedup

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_dedup_tool: make SampleDedupGlobal::sampling_rate const

Clarifies why it doesn't need to be protected by a lock.

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_dedup_tool: remove unused SampleDedupGlobal::object_dedup_threshold

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_dedup_tool: fix unnecessarily complicated bool returns

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_dedup_tool: SampleDedupWorkerThread can simply inherit from Thread

SampleDedupWorkerThread didn't actually use any of the facilities on
CrawlerThread and indeed left several of them uninitialized.

Also allows us to remove the CrawlerThread constructor which left
several members uninitialized.

Signed-off-by: Samuel Just <sjust@redhat.com>

qa: remove unused values in deduplication.py

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

qa: add validation stage for deduplication.py

To validate sample-dedup actually works, validate() runs
separated thread from sample-dedup and verifies
two following things.
1. check sample-dedup starts properly.
2. check references of all the chunk objects' in chunk tier
   exists in designated base pool.
This routune repeats for max_valication_cnt times while
sample-dedup is running. If it doesn't raise any fail while the loop,
we can pretend sample-dedup works accurately.
If not, assert() will stop this test.

In case that a reference of chunk object doesn't exist in base pool,
validate() gives a second chance after repairing it (chunk-repair op)
to deal with false-positive reference inconsistency.

Signed-off-by: Sungmin Lee <sung_min.lee@samsung.com>

src/tools/ceph_dedup_tool: fixes to daemonize the process correctly

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

qa: add deduplication test under RGW workload (s3)

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #48157 from zdover23/wip-doc-2022-09-18-rados-network-config-prompt-correction

doc/rados: update prompts in network-config-ref

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: update prompts in network-config-ref

This PR adds unselectable prompts to network-config-
ref.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

rgw: remove unused AioThrottles in rgw_op.cc

throttle creation was moved to RadosStore::get_atomic_writer(),
RadosStore::get_append_writer(), and RadosMultipartUpload::get_writer()

these instances in rgw_op.cc were unused

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47849 from torchiaf/fix/37327

mgr/dashboard: Add details to the modal which displays the `safe-to-d…

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #47958 from rhcs-dashboard/edit-host-labels-fix

mgr/dashboard: fix Expected to find element: `cd-modal .badge but never found it

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

win32: skip installing openssl man pages

This speeds up the build a bit and makes the logs easier to read.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #45148 from guits/revert-ceph-crash-changes

Revert "src/ceph-crash.in: remove unused frame in handler()"

Reviewed-by: Teoman ONAY <tonay@redhat.com>

Merge pull request #47965 from cbodley/wip-rgw-drain-cb-capture-error

Revert "rgw: add coroutine drain callback capture_error()"

Reviewed-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

mgr/dashboard: Add details to the modal which displays the `safe-to-destroy` result

- Add warnings type information in the case of the OSDs are not safe to destroy
- Add info type information in the case of the OSDs are safe to destroy

Fixes: https://tracker.ceph.com/issues/37327
Signed-off-by: Francesco Torchia <francesco.torchia@suse.com>

Merge pull request #47878 from xxhdx1985126/wip-pin_set-cpu

crimson/os/seastore/btree: reduce the avg number of queries of pin sets' parent/child lookup

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

build: Remove -fno-new-ttp-matchingg flag

This was added in the upgrade to C++17. It's no longer needed since
fixing Clang compatibility got rid of non-conforming templates.

It's no longer needed and getting rid of it is a (minor) quality of
life enhancement since it gets rid of a spurious error when using
Clang based build tools (language server, etc.) while compiling with
GCC.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #47090 from lxbsz/wip-56553

client: do not uninline data for read

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #48037 from guits/fix_57085

ceph-volume: fix inventory with device arg

crimson/osd: initialize PGAdvanceMap::from inside critical section

Fixes: https://tracker.ceph.com/issues/57542
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #48117 from guits/fix-cv-func-test-filestore-create

ceph-volume/tests: fix lvm centos8-filestore-create job

Merge pull request #48054 from yuvalif/wip-yuval-fix-56572

tests: remove pubsub tests from multisite

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kalpesh Pandya <kapandya@redhat.com>

Merge pull request #47516 from nmshelke/fix-56632

qa: ignore disk quota exceeded failure in test

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

ceph-volume/tests: fix lvm centos8-filestore-create job

/dev/vdc1 can't be zapped if it still holds an lv mapper.
let's use --destroy in the lvm zap command in order to remove
the held lv mapper before zapping the partition and recreate the partition after.

Fixes: https://tracker.ceph.com/issues/57553
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>