Removed pandas from requirements.txt, ceph.spec.in, and debian/control
because of installation issues in RHEL/CentOS.
Replaced pandas usages in RHDiskFailurePredictor with similar numpy
counterparts (e.g. structured array instead of dataframe)
Replaced joblib usages with pickle because older version of scikit-learn
did not list joblib as a dependency and so it wasnt getting installed.
Using joblib would have required specifying it as a separate dependency
in spec file and requirements.
Karanraj Chauhan [Wed, 30 Oct 2019 16:17:35 +0000 (12:17 -0400)]
mgr/diskprediction_local: Updated dependencies in ceph.spec.in, debian/control to match requirements.txt
Added pandas dependency to ceph.spec.in and debian/control.
In the spirit of "if it aint broke, dont fix it", I did NOT add
scikit-learn as a dependency in spec or control, because scikit-learn
was already a dependency in diskprediction_local, and so it should have
already have been taken care of.
Also in the same spirit, removed joblib dependency from requirements.txt
because scikit-learn depends on it and therefore joblib will get
installed when scikit-learn gets installed.
Karanraj Chauhan [Tue, 15 Oct 2019 15:30:52 +0000 (11:30 -0400)]
mgr/diskprediction_local: Updated Red Hat developed prediction model. Updated module options to choose between Red Hat and ProphetStor models.
predictor.py contains definition for the original ProphetStor developed model as well
as Red Hat developed model (using Backblaze dataset).
User can choose which model to use by passing either 'prophetstor' or 'redhat' to the
`Module`'s `predictor_model` config option.
Updated the disk health data formatting code in `Module` to include `user_capacity`,
`vendor`, etc fields that are used by the RHDiskFailurePredictor. These will simply
be ignored by the PSDiskFailurePredictor
Updated preprocessing in RH model to use the data passed from module directly instead
of restructuring again. Added logging instead of print statements.
Restructured pretrained models directory to accomodate both models files.
mgr/diskprediction_local: Updated prediction models to use only supported python packages.
Removed non-supported python packages from requirements.txt
Added scikit-learn based models, removed rgf-python based models.
Updated config.json and DiskPredictor.__preprocess for the same.
Also added manufacturer as argument to DiskPredictor.__preprocess
Updated manufacturer lookup - first check if available as smartctl field,
if not then try to infer from model name.
Updated predicted class to be the prediction for the most recent day in
time series data given.
Updated naming convention from "preprocessor" to "scaler".
mgr/diskprediction_local: Replaced old models and updated predictor.
ProphetStor models are replaced with in-house developed models.
Preprocessors are also stored in addition to the prediction models.
Objects are now stored using joblib instead of pickle, as recommended by
scikit-learn docs.
"manufacturer-specific" models are used instead of "best-feature-match"
models. i.e., instead of models being trained (presumably) just based on what
features are available, models have been trained for each manufacturer.
This is because of variation in meaning and availibility of SMART
attributes across manufacturers.
Updated config.json, requirements.txt, and DiskFailurePredictor for these changes.
Merge pull request #31070 from sebastian-philipp/dashbaord-run-backend-zsh
mgr/dashboard: Fix zsh support in run-backend-api-tests.sh
Reviewed-by: Alfonso Martínez <almartin@redhat.com> Reviewed-by: Laura Paduano <lpaduano@suse.com> Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Sage Weil [Wed, 6 Nov 2019 15:46:36 +0000 (09:46 -0600)]
Merge PR #31440 into master
* refs/pull/31440/head:
qa/standalone/test_ceph_daemon: Make container images configurable
qa/standalone/test_ceph_daemon: Allow running from root dir
qa/standalone/test_ceph_daemon: Fix hang when CEPH_DAEMON is not set
Sage Weil [Wed, 6 Nov 2019 15:33:36 +0000 (09:33 -0600)]
Merge PR #31385 into master
* refs/pull/31385/head:
mgr/ssh: add 'osd rm'
mgr/ssh: keep inventory and service cache keys in sync
mgr/orchestrator_cli: fix 'service ls' table format
mgr/ssh: invalidate service state on create, remove, and service action
mgr/orchestrator: add invalidate() to OutdatableDictMixin
mgr/ssh: cache services
mgr/ssh: drop specialized _remove_{mds,mgr,rgw}
mgr/ssh: change inventory_cache_timeout to seconds (not minutes)
mgr/orchestrator: make wait poll every 1s (not 5s)
Sage Weil [Mon, 4 Nov 2019 19:43:36 +0000 (13:43 -0600)]
mgr/ssh: keep inventory and service cache keys in sync
Our host list is implicitly defined as the set of keys for the inventory
(and service) caches. Ensure they are in sync by unioning and inserting
any missing items.
Sage Weil [Tue, 5 Nov 2019 14:55:59 +0000 (08:55 -0600)]
Merge PR #31303 into master
* refs/pull/31303/head:
ceph-daemon: make systemd unit descriptions Capitalized
ceph-daemon: handle daemon names with multiple .'s
mgr/ssh: rgw: ensure map rgws are mapped to the zone
mgr/ssh: use . as delimiter between service group and id
mgr/ssh: implement 'rgw update'
mgr/orchestrator_cli: add 'rgw update' CLI command
mgr/ssh: add 'rgw rm'
mgr/ssh: implement 'rgw add'
ceph-daemon: handle 'rgw' type daemons
mgr/orchestrator_cli: rgw add: optionally take count and node names
Add generic CLI commands to turn device lights on and off.
These will call out to orchestrator stubs, once it has the appropriate hook
implemented, which will in turn call out to salt or rook or ssh or whatever
to actually turn a light on or off.
Nathan Cutler [Thu, 24 Oct 2019 11:26:50 +0000 (13:26 +0200)]
ceph-backport.sh: implement interactive setup routine and new options
This commit implements several new features:
* a --cherry-pick-only option
* a --force option
* an --existing-pr option
* an interactive setup routine
The --cherry-pick-only option can be used to test whether a backport
cherry-picks cleanly, for example. This is the same as the --prepare
functionality that was provided by an earlier version of the script, and
--prepare is re-introduced as a synonym for --cherry-pick-only.
The --force option can be used to make the script less careful (less
"cowardly"). For example, if the script refuses to do a backport because the
backport tracker issue is assigned to someone else, the script will "cowardly"
refuse to continue. Use --force to override. Be aware that --force will also
blow away an existing wip branch - the script asks for user confirmation in this
case.
The new --existing-pr option can be used to specify the number (ID) of an
existing backport PR that addresses the backport tracker issue given via the
positional argument.
The new "interactive setup routine" should make the setup process much simpler
for the user. If there is a setup issue, the script produces a report and starts
the interactive setup routine, which prompts the user for the needed
information.
Also, the script no longer requires the user to explicitly provide values for
github_user and redmine_user_id. Instead, it divines the correct values from the
GitHub token and the Redmine key, respectively.
Finally, the existing ~/bin/backport_common.sh file is deprecated in favor of
two files:
~/.github_token
~/.redmine_key
(The latter is already used by Sage's build-integration-branch tool and it
didn't make sense to have two different configuration files for a single
purpose.)
projects / ceph.git / log