chunmei Liu [Thu, 22 Aug 2019 07:47:40 +0000 (00:47 -0700)]
crimson: add FuturizedCollection
we will need to talk to BlueStore, and currently `ceph::os::Collection`
only works with `CyanStore`, so a wrapper around `ceph::os::Collection`
is added. we can subclass it to adapt to CyanStore and to BlueStore.
the underlying concrete `FuturizedStore` classes are responsible to
cast `FuturizedCollection` to their own `Collection` type if they
want to access the interface not exposed by `FuturizedCollection`.
Signed-off-by: chunmei Liu <chunmei.liu@intel.com>
Kefu Chai [Thu, 22 Aug 2019 08:47:19 +0000 (16:47 +0800)]
crimson/osd: lift continuation in PGMeta::load()
after making `open_collection()` return a future, we don't need to chain
the body of `PGMeta::load()` after `omap_get_values()`. instead we can
lift it.
Nathan Cutler [Wed, 21 Aug 2019 14:49:26 +0000 (16:49 +0200)]
script: add backport-resolve-issue
This script processes GitHub backport PRs, checking for proper cross-linking
with a Redmine Backport tracker issue and, if a PR is merged and properly
cross-linked, it can optionally resolve the tracker issue and correctly
populate the "Target version" field.
The script takes a single positional argument, which is optional. If the
argument is an integer, it is assumed to be a GitHub backport PR ID (e.g. "28549").
In this mode ("single PR mode") the script processes a single GitHub backport
PR and terminates.
If the argument is not an integer, or is missing, it is assumed to be a
commit (SHA1 or tag) to start from. If no positional argument is given, it
defaults to the tag "BRI", which might have been added by the last run of the
script. This mode is called "scan merge commits mode".
In both modes, the script scans a local git repo, which is assumed to be
in the current working directory. In single PR mode, the script will work
only if the PR's merge commit is present in the current branch of the local
git repo. In scan merge commits mode, the script starts from the given SHA1
or tag, taking each merge commit in turn and attempting to obtain the GitHub
PR number for each.
For each GitHub PR, the script interactively displays all relevant information
(NOTE: this includes displaying the GitHub PR and Redmine backport issue in
web browser tabs!) and prompts the user for her preferred disposition.
Kefu Chai [Tue, 27 Aug 2019 01:22:34 +0000 (09:22 +0800)]
install-deps.sh: only prepare wheels for "make check"
* move `for_make_check` to the beginning of script, as FreeBSD will also
use this variable
* extract `preload_wheels_for_tox()` function out to improve readability
* call `preload_wheels_for_tox()` only if `for_make_check` is true
When --resolve-parent is provided on the command line, the script
will check the status of backport issues for those parent issues
all of whose backport issues already exist. If all the backport
issues are in status "Resolved", the parent issue's status is set
to "Resolved" as well.
Kefu Chai [Sun, 25 Aug 2019 13:22:16 +0000 (21:22 +0800)]
common/DecayCounter: do not decay when testing denc
check-generated.sh tries to verify the invariance of a type after
encoding and decoding. so we should keep the value of `DecayCounter` the
same when encoding/decoding/dumping.
we should update explicit loc and location even if
`ofs >= manifest->obj_size`. there is chance that we are updating an end
iterator whose ofs is equal to obj_size. before being updated, the end
iterator points to an implicit location, while after being updated, the
manifest could be using an explicit location, so we should update the
end iterator as well.
Kefu Chai [Mon, 26 Aug 2019 03:36:50 +0000 (11:36 +0800)]
install-deps.sh: download wheel using 'pip wheel'
otherwise we will fail to install the build dependencies of
`lazy-object-proxy` from the wheelhouse. as `lazy-object-proxy` does not
add `setuptools_scm` in its `setup.py`, instead it lists
`setuptools_scm` in `setup.cfg` and `pyproject.toml` as a `build-system`
requires. but unfortunately, `pip download` only downloads the
install/run-time dependencies at this moment. and `lazy-object-proxy`
does not offer binary package for at least python2.7.
ideally, `pip download` should collects its dependencies like
xie xingguo [Mon, 26 Aug 2019 02:28:34 +0000 (10:28 +0800)]
osd/osd_types: add_next_event - add special handler for lost_revert
For unfound objects, we might append LOST_REVERT log entries,
which shall allow these objects to be reverted to the newest
available version later.
However, we are currently lack of support to rewind the clean_regions
portion too when marking unfound objects as lost with inc-recovery mode
enabled. Hence we must mark these unfound-revert objects as fully dirty
to make sure they can be correctly recovered.
E.g.,:
- primary is pulling object A from replica 1
- object A is corrupted on replica 1
- object A is now unfound
- mark object A as lost, replica 1 will persist a wrong
missing item for object A..
xie xingguo [Sat, 24 Aug 2019 01:51:20 +0000 (09:51 +0800)]
osd/osd_types: drop 'new_object' from constructor
There is no consumer.
Actually, I think this field is only meaningful to be used
to indicate whether we should initiate an inc-recovery or not.
If not, then we shall fall back to triggering a full-recovery
instead.
xie xingguo [Sat, 24 Aug 2019 02:56:57 +0000 (10:56 +0800)]
osd/osd_types: always call mark_fully_dirty for missing.add
In general we shall build missing set (and hence clean_regions)
based on pg log. However, currently there are still 5 cases we might call
missing.add to add a new pg_missing_item into the missing set
explicitly (or replace an existing pg_missing_item entirely):
1. we explicitly build missing set on startup, in which case
we know we are trying to be compatiable with pre-kraken versions,
so it should be ok for us to disable inc-recovery.
2. we are currently processing authoritative log, and there are
some divergent objects detected. For simplicity (and correctness),
we should disable inc-recovery entirly for these objects.
3. we are re-building missing set, e.g., due to the global
CEPH_OSDMAP_RECOVERY_DELETES policy changing.
In this case we know we are at the end of upgrading from a
pervious version that is lack of CEPH_OSDMAP_RECOVERY_DELETES support.
Hence it should be the recommended option to disable inc-recovery
simultaneously since these objects should be lack of inc-recovery support
too.
4. we are adding or re-adding missing object into primary's missing_loc.
It doesn't matter whether we have a correct clean_regions there
since we never actually refer to that field from missing_loc
when we actually start to perform object recovery later.
5. we are auto-repairing a corrupted object and hence the need of
adding it to the corresponding missing set first, e.g, by leveraging
the existing recovery procedure. In this case, we always disable
inc-recovery to make sure this object can be fully (and correctly)
recovered later.
Do this outside the standard tick interval as it needs to be driven more
frequently to keep up with client workloads that generate a lot of
capabilities.
Fixes: https://tracker.ceph.com/issues/41141 Fixes: https://tracker.ceph.com/issues/41140 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
crimson/osd: introduce OpsExecuter to uniform calling CEPH_OSD_OPS.
OSD has two entry points for executing CEPH_OSD_OP_*:
1. the MOSDOp message handler,
2. the Object Class API (cls_* and cls_cxx_* functions).
We definitely want to address these two users without code
duplication. However, exposing the entire PG to Obj Class
would break encapsulation. Moreover, there is difference
in life times between PG and sequence-of-operations-from-
MOSDOp.
crimson/osd: exceptions derive from std::system_error now.
This change is be useful especially for CEPH_OSD_OP_CALL
which will be brought by further commits. The issue here
is that the Ceph Classes will be based on existing iface
handling errors with usual, int-based ret codes. These
codes needs to be glued with the existing handling in
pg::do_osd_ops().
projects / ceph-ci.git / log