mgr/dashboard: avoid overriding the configuration in package.json

the warning in npm run build is `Option "configuration" was already specified with value "en-US". The
new value "production" will override it.`

This causes the dist for production env to be created in dist/ whereas
it should be dist/en-US (or whatever npm_config_locale. That prevented
from dashboard to load and cause the cephadm e2e checks to fail.

https://jenkins.ceph.com/job/ceph-dashboard-cephadm-e2e-nightly-master/185/consoleFull#57532618373a8703b-5adb-41c5-84a0-8cf4e065ba3d

Related to https://github.com/ceph/ceph/pull/46143

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #46164 from rishabh-d-dave/vstart-minor-fixes

qa/vstart_runner: minor bug fix and minor adjustments

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46217 from Matan-B/wip-matanb-c-shard_services

crimson/osd: Moving shard_services to PGBackend

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #46208 from rzarzynski/wip-crimson-main-using-cleanups

crimson/osd: drop "using ..." from main.cc

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #46110 from dparmar18/fix_onecmd

cephfs-shell: make onecmd() print proper error msg

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46047 from dparmar18/qa_testcephfs_shell_pep8_fixes

qa/cephfs/test_cephfs_shell: Fixed pep8 formatting violations

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45808 from rhcs-dashboard/add-cpu/mem-usage

mgr/dashboard: introduce memory and cpu usage for daemons

Reviewed-by: Sarthak0702 <NOT@FOUND>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: ceph-jenkins <NOT@FOUND>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: sunilangadi2 <NOT@FOUND>

Merge pull request #46190 from rhcs-dashboard/smart-data-fix

mgr/dashboard: fix smart data error

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #45894 from rishabh-d-dave/stop-sh-check-mdss

src/stop.sh: check if ceph-mds daemon(s) exist beforehand

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46143 from tchaikov/wip-dashboard-angular

cmake: s/--prod/--configuration=production/

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

crimson/osd: Moving shard_services to PGBackend

ec and replicated backends are derived from PGBackend,
shard_services should be a member of the base class.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #46093 from pdvian/wip-admin-curl

examples: Add rgw script to access admin APIs using curl

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #46196 from rhcs-dashboard/smart-data-scsi

mgr/dashboard: smart data for devices with scsi protocol

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #46201 from rhcs-dashboard/daemon_count_hosts

mgr/dashboard: show daemon count instead of services in host page

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: sunilangadi2 <NOT@FOUND>

Merge pull request #46174 from rhcs-dashboard/physical-disks-selection

mgr/dashboard: devices with same UID causes multiselection

Reviewed-by: Sarthak0702 <NOT@FOUND>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #46159 from rhcs-dashboard/disable-external-grafana-snapshots

mgr/cephadm: Disable external snapshots in grafana

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #46169 from rishabh-d-dave/vr-run-mnt-cmd

qa/vstart_runner: fix bug in LocalFuseMount._run_mount_cmd()

Reviewed-by: Venky Shankar <vshankar@redhat.com>

mgr/dashboard: smart data for devices with scsi protocol

In the dashboard, we've been showing smart data for hdd devices with ata
protocol only. Otherwise we show a No Smart Data found error which is
clearly misleading since Smart Data is returned even in the api call.

So this PR is trying to show the smart data for hdd devices
that uses scsi protocol too.

Fixes: https://tracker.ceph.com/issues/55574
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: show daemon count instead of services in host page

Fixes: https://tracker.ceph.com/issues/47218
Signed-off-by: Nizamudeen A <nia@redhat.com>

qa/vstart_runner: fix args to os.path.join()

Result of os.path.join() before "./bin/ceph-mds" and after
"./bin/./ceph-mds".

Before -

2022-05-05 19:36:11,100.100 DEBUG:__main__:> ./bin/./ceph-mds -i a

After -

2022-05-05 19:38:48,179.179 DEBUG:__main__:> ./bin/ceph-mds -i a

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/vstart_runner: minor bug fix and minor adjustments

The message regarding deletion of helper tools is printed for every
command. This message should be printed only when applicable.

Besides -

* Move XXX comments to _do_run() since it increases visibility of
  these messages.
* Move omission of arguments stuff to new method clear up the clutter.
* And remove shell as a parameter from _perform_checks_and_adjustments
  since it's redundant.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #37667 from rishabh-d-dave/vr-inherit-Remote

vstart_runner: let LocalRemote inherit Remote

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46194 from athanatos/sjust/wip-55565

crimson: fix op tracking related bugs

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/common/operation: trim AggregateBlockingEvent::events entries

PglogBasedRecovery and BackfillRecovery reuse the same Operation
until their respective operations are complete.  Each recovery
operation adds an entry to AggregateBlockingEvent::events.  This
way, we only retain entries that are currently blocking.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #46127 from Matan-B/wip-matanb-cyan-clone

crimson/os: Add OP_CLONE to cyanstore

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #46205 from rzarzynski/wip-crimson-cls_rgw-objapi

crimson/osd, rgw: implement the new Objclass API methods for cls_rgw

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44418 from AmnonHanuhov/wip-fix_futurized_store

crimson/os: Don't limit the amount of returned keys per omap get call

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd: drop "using std::string" from main.cc

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: drop "using config_t = ..." from main.cc

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd, rgw: implement the new Objclass API methods for cls_rgw

They were introduced by 3877c1e37f2fa4e1574b57f05132288f210835a7.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #45732 from mchangir/mgr-snap_schedule-validate-fs-for-all-commands

mgr/snap_schedule: validate fs name argument

Reviewed-by: Venky Shankar <vshankar@redhat.com>

cephfs-shell: make onecmd() print proper error msg

Rationale: Whenever a python exception occurred in cephfs-shell,
           it would often only be the exception message but doesn't
           say anything about the type of exception. For example if
           `ZeroDivisionError: division by zero` occurred, the onecmd()
           would print `division by zero` but will omit the type of
           exception. In this case it's easy to understand but let's
           say an `KeyError` exception occurred for a key `9999` which
           is not existent in the dictionary, onecmd() would print
           just `9999` in this scenario and it would be very difficult
           to interpret what type of error it is.

Fixes: https://tracker.ceph.com/issues/55536
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

Merge branch 'master' into qa_testcephfs_shell_pep8_fixes

qa/cephfs/test_cephfs_shell: Fixed pep8 formatting violations
NOTE: Although most of the issues are fixed but a few function
      and variable names are unchanged in order to prevent
      ambiguity and preserve their meaning.

      They are:
      - functions: setUp(), test_ls_H_prints_human_readable_file_size(),
      - variables: ls_H_output, ls_H_file_size

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

mgr/dashboard: fix smart data error

the error in the log was this

```
"/usr/share/ceph/mgr/dashboard/services/ceph_service.py", line 253, in _get_smart_data_by_device
May 06 07:38:39 occldlr750-1.occl208.lab conmon[2142938]:     svc_type, svc_id = daemon.split('.')
May 06 07:38:39 occldlr750-1.occl208.lab conmon[2142938]: ValueError: too many values to unpack (expected 2)
```

on the cluster, the output of `ceph device ls-by-host` looks like this

```
ceph: root@occldlr750-1 /]# ceph device ls-by-host occldlr750-1.occl208.lab
DEVICE                        DEV   DAEMONS                       EXPECTED FAILURE
DELLBOSS_VD_cbd004c975390010  sda   mon.occldlr750-1.occl208.lab
WDC_WUH721818AL5204_3FGZR3JT  sdda  osd.20
WDC_WUH721818AL5204_3FH4315T  sdbf  osd.94
WDC_WUH721818AL5204_3FHP58TT  sdec  osd.30
WDC_WUH721818AL5204_3FHSK8HT  sdu   osd.78
WDC_WUH721818AL5204_3FHVTS9T  sdfi  osd.47
WDC_WUH721818AL5204_3FHWJE8T  sdv   osd.23
WDC_WUH721818AL5204_3FHXHETT  sdcl  osd.11
WDC_WUH721818AL5204_3FHXKP1T  sdcj  osd.10
```

the first device is mon and its name is mon.occldlr750-1.occl208.lab.

In our dashboard code, when fetching the smart data we have a line like
this

`svc_type, svc_id = daemon.split('.')`

so for the mon the output of `daemon.split('.') will be ['mon', 'occldlr750-1', 'occl208', 'lab']. The svc_id gets split into three because of the split. I am changing that and giving the criteria as splitting only on the first occurence of the dot and the considering everything that comes after the dot as the svc_id of the device.

Fixes: https://tracker.ceph.com/issues/55571
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #46064 from liu-chunmei/crimson-pool-quota

Crimson pool quota

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/dashboard: devices with same UID causes multiselection

In the Physical Disks page, the uids for multiple devices are coming in
as same and that causes the selection to go berserk and select multiple
rows with same UID. The uid is generated in the frontend service call
itself. I just added some more parameters to it inorder to make it more
unique.

The second issue is the number of selected number getting multiplied
exponentially. Its because each time the table is updated or refreshed,
we push the row with the number of selected items we had before and that
causes the number of selection to multiply.

Fixes: https://tracker.ceph.com/issues/55523
Signed-off-by: Nizamudeen A <nia@redhat.com>

crimson/os: Add OP_CLONE to cyanstore

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd: check full

need check if pool exceeds its quota

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson/errorator: change errorator template to int

need define EDQUOT which is not included in std::errc.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson/osd/pg_recovery: remove new_started, fix parallel_for_each call

Presumably typo from b7e2b11595c6e2cd2c638135ef6f044c40efaa87.

Fixes: https://tracker.ceph.com/issues/55565
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/operation: use std::list for AggregateBlockingEvent::events

std::vector's references aren't stable after emplace_back.

Fixes: https://tracker.ceph.com/issues/55565
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/pg_recovery: avoid using create_part_trigger() directly

AggregateBlockingEvent::TriggerI::maybe_record_blocking() should handle
invoking create_part_trigger().

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: retain reference to WaitForObjectRecovery until blocker resolves

PGRecovery::on_global_recover destroys the map entry without waiting for
the future returned from

    seastar::future<> wait_for_recovered(BlockingEvent::TriggerI& trigger) {

This commit changes WaitForObjectRecovery to be refcounted and retains a
reference until the future resolves.

Fixes: https://tracker.ceph.com/issues/55565
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #46170 from elacunza/patch-1

doc/rados/operations: Add missing required parameters to stretch-mode CRUSH rule example

examples: Add rgw script to access admin APIs using curl

The rgw_admin_curl.sh script will allow end-user/developers to
access RGW admin APIs through curl command.

Signed-off-by: Prashant D <pdhange@redhat.com>

Merge pull request #46108 from rhcs-dashboard/fix-ingress-ssl-cert-validation

mgr/dashboard: fix ssl cert validation for ingress service creation

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: sunilangadi2 <NOT@FOUND>

mgr/cephadm: Disable external snapshots in grafana

This PR intends to disable the external snapshots in grafana to reduce security threats

Signed-off-by: Aashish Sharma <aasharma@redhat.com>

doc/rados/operations: Add missing required parameters to stretch-mode CRUSH rule example

Rule example is missing min_size and max_size, both are required.

Signed-off-by: Eneko Lacunza <elacunza@binovo.es>

crimson: remove unused PGRecovery method declarations

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #45935 from dparmar18/cephfs-shell-doc-update

doc/man/cephfs-shell.rst: add use of exclamation mark with cmds

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46097 from tchaikov/wip-crimson-cleanups

crimson/osd/main: s/values/config_values/

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #45788 from rzarzynski/wip-crimson-op-tracking-2

crimson: bring the core of op tracking

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #46006 from phlogistonjohn/jjm-nfs-cleanups-pt4

mgr/nfs: minor organizational cleanups

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #46035 from rkachach/fix_issue_52906

mgr/cephadm: adding logic to close ports when removing a daemon

Reviewed-by: Adam King <adking@redhat.com>

qa/vstart_runner: fix bug in LocalFuseMount._run_mount_cmd()

This methods fails to collect return value from
FuseMount._run_mount_cmd() and return it. This leads to a bug for tests
that expect mount command to fail when executed with vstart_runner.py.

Fixes: https://tracker.ceph.com/issues/55553
Signed-off-by: Rishabh Dave <ridave@redhat.com>

crimson/osd: drop legacy base classes for pipeline stages

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: drop the old blocking infrastructure

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: strip blokcing_future from OperationT

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate OperationThrottler to new tracking infra.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: crimson/osd: drop blocking_future from PGActivationBlocker

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: crimson/osd: drop blocking_future from PGMap

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: drop blocking_future from OSDMapGate

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson: bring the WaitBarrier tracking

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

vstart_runner: make LocalRemote inherit RemoteShell

And therefore get rid of methods duplicated in LocalRemote and add a
call to empty constructor of RemoteShell in LocalRemote.__init__().

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #46033 from iqbalredkhan/quota2

rgw: Refactoring Quota Management

crimson: move PipelineStageIT::enter() dispatch to compile-time

The optimization isn't the goal (but rather a side effect). The
reason behind this change is to let pipeline stages define exit
barriers incorporating `OpT`-dependant `Triggers` to track blocking
that happens on the barriers.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/os: add junction between OSD's ops and Seastore's OrderingHandle.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate AggregateBlocker-related ops to new tracking infra

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: bring AggregateBlockingEvent.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate PeeringEvent to new tracking infra, part 2

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: add BlockingEvent-aware variant of OSD::get_or_create_pg

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate InternalClientRequest to new tracking infra.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate RecoverySubRequest to new tracking infra.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate RepRequest to new tracking infra

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate PGAdvanceMap to new tracking infra.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: reduce unnecessary indentation level in PGAdvanceMap::start()

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate PeeringEvent to new tracking infra.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate BackgroundRecovery to new tracking infra.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate BackfillRecoveryPipeline to new tracking infra.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: CompoundPeeringRequest tracks its start and completion too

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate CompoundPeeringRequest into new tracking infra

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: move SubOpBlocker from .cc to header

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: PGActivationBlocker does support new tracking infra

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: dissect PG::WaitForActiveBlocker into PGActivationBlocker

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: PGMap blocker does support new tracking infra

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: make PhaseOperationT able to deal with interruptors

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: pg_map.h doesn't pull pg.h anymore

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: convert entire ClientRequest to the new infra

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: slightly optimize OperationRegistryT::do_register()

There is no need to call the virtual op::get_type() twice.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: fix an assertion failure in Formatter when dumping ops

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: implement dump_ops_in_flight

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: track start and completion of ClientRequest

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: bring boilerplate for LTTng-based external tracking backend

At the moment it's mostly for verification & demonstration purposes.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: drop forward declaration of OSDMap osd.h

Having it doesn't make sense as we already include the header.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: add registry of external handlers of tracking events

This commit intentionally fails the build. It does that to verify
the static assertion on per-op-type lookup of event registry.

The concrete assertion is the one about `ClientRequest` as we
already moved two of its blocker into the new op tracking infra.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: switch one OSDMapGate of ClientRequest to new blocking infra

This time the demo of `BlockerT::BlockingEvent::TriggerI` which
is supposed to handle situations where:

1) `BlockerT` is separated from `TrackableOperationT` by many
intermediaries while
2) we don't want to templatize them all.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: switch a stage of ClientRequest to new blocking infra

This is a debut of the new blocking & tracking infrastructure
using the type-aware `BlockerT::BlockingEvent::Trigger<T>`.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: introduce PhasedOperationT to deduplicate PipelineHandles

This new class in the hierarchy will be handy when it will come to
introduce helpers for deailing with entering a pipeline stage.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: switch all OSD operations to TrackableOperationT

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>