gal salomon [Mon, 12 Apr 2021 05:54:37 +0000 (08:54 +0300)]
parquet implementation:
(1) adding arrow/parquet to make(install is missing)
(2) s3select-operation contains 2 flows CSV and Parquet
(3) upon parquet-flow s3select processing engine is calling (via callback) to get-size and range-request, the range-requests are a-sync, thus the caller is waiting until notification.
(4) flow : execute --> s3select --(arrow layer)--> range-request --> GetObj::execute --> send_response_data --> notify-range-request --> (back-to) --> s3select
(5) on parquet flow the s3select is handling the response (using call-backs) because of aws-response-limitation (16mb)
add unique pointer (rgw_api); verify magic number for parquet objects; s3select module update
fix buffer-over-flow (copy range request)
change the range-request flow. now,it needs to use the callback parametrs (ofs & len) and not to use the element length
refactoring. seperate the CSV flow from the parquet flow, a phase before adding conditional build(depend on arrow package installation)
adding arrow/parquet installation to debian/control
align s3select repo with RGW (missing API"s, such as get_error_description)
undefined reference to arrow symbol
fix comment: using optional_yield by value
fix comments; remove future/promise
s3select: a leak fix
s3select: fixing result production
s3select,s3tests : parquet alignments
typo: git-remote --> git_remote
s3select: remove redundant comma(end of projections); bug fix in parquet flow upon aggregation queries
adding arrow/parquet
editorial. remove blank lines
s3select: merged with master(output serialization,presto alignments)
merging(not rebase) master functionlities into parquet branch
(*) a dedicated source-files for s3select operation.
(*) s3select-engine: fix leaks on parquet flows, enabling allocate csv_object and parquet_object on stack
(*) the csv_object and parquet object allocated on stack (no heap allocation)
move data-members from heap to stack allocation, refactoring, separate flows for CSV and parquet. s3select: bug fix
conditional build: upon arrow package is installed the parquet flow become visable, thus enables to process parquet object. in case the package is not installed only CSV is usable
Sage Weil [Thu, 6 Jan 2022 13:54:45 +0000 (08:54 -0500)]
Merge PR #44054 into master
* refs/pull/44054/head:
doc/rados/operations: document pg_num_max
mgr: set max of 32 pgs for .mgr pool
mgr/dashboard: expect pg_num_max property for pools
mon/OSDMonitor: add option --pg-num_max arg for create pool
mon/OSDMonitor: disallow setting pg_num < min or > max
mgr/pg_autoscaler: apply pg_num_max
mon: add pg_num_max pool property
Reviewed-by: Neha Ojha <nojha@redhat.com> Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
John Bent [Wed, 5 Jan 2022 16:04:40 +0000 (09:04 -0700)]
README.md: Update README.md to add link to tracker.ceph.com
I searched the existing documentation for a link to the tracker and had a hard time finding it. Other folks like myself might like it so prominently displayed.
Added a link to https://tracker.ceph.com/projects/ceph as https://tracker.ceph.com/ is basically blank and not totally intuitive how to get to issues.
Matt Benjamin [Wed, 22 Dec 2021 18:33:52 +0000 (13:33 -0500)]
rgwlc: harmonize lifecycle notifications with the others
After working with notification configurations, it seemed to make
more sense to organize the different lifecycle notification sub-types
along the same lines as the official AWS ones. So, there are now
ObjectExpirationCurrent and ObjectExpirationNoncurrent types, and
an ObjectExpiration wildcard. Similarly, ObjectTransition
contains Current and Noncurrent sub-types.
Note that I have defined an ObjectExpirationAbortMPU notification
sub-type, but do not currently generate it--this is to avoid changing
the rgw::sal interface, but most likely we would support this in
future.
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Matt Benjamin [Tue, 26 Jan 2021 17:52:47 +0000 (12:52 -0500)]
rgwlc: optionally support notifications on object expiration
Most of the work is to remove direct knowledge of req_state from
methods in rgw_notify.
I've chosen to create new notification types matching the different
expire actions (but not transition). The new event types are not
nested under Delete. Notifications are sent iff rgw_lc_notify is true
(default false).
Adjusted per comments in initial review, in particular, notification from
lifecycle is no longer conditional on a config setting, and constness
is restored.
Fixes: https://tracker.ceph.com/issues/49068 Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Nizamudeen A [Thu, 30 Dec 2021 08:28:58 +0000 (13:58 +0530)]
mgr/dashboard: stabilizing the cephadm dashboard e2e
Reordering the tests and adding some more tests to verify the cluster is
healthy before proceeding to do some complex tasks like maintenance and
drain host
Fixes: https://tracker.ceph.com/issues/53742 Signed-off-by: Nizamudeen A <nia@redhat.com>
Venky Shankar [Mon, 13 Dec 2021 06:15:19 +0000 (01:15 -0500)]
mds: ignore unknown client op when tracking op latency
Server::handle_client_request() ignores unknown client operation
by returning -ENOTSUPP, however, Server::perf_gather_op_latency()
aborts on unknown client op, thereby causing -ENOTSUPP to never
reach the client. ceph_abort() seems unnecessary here.
Note, we could have invoked Server::perf_gather_op_latency()
when the return value to client is not -ENOTSUPP, however,
a valid client operation *might* just return -ENOTSUPP in
some cases.
@mchangir ran into this with his getvxattr op changes (PR #42001).
Yin Congmin [Fri, 3 Dec 2021 03:07:52 +0000 (11:07 +0800)]
librbd/cache/pwl/: remove IO waited state
IO waited state variables including waited_lanes, waited_entries,
waited_buffers are only used to print logs and have no practical
effect.no_space = true is enough to indicate state.
Yin Congmin [Wed, 1 Dec 2021 06:37:50 +0000 (14:37 +0800)]
doc/rbd: change persistent write-back cache to persistent write log cache
The full name of PWL is persistent write log, and the full name of
RWL is replica write log. Change the title to make it consistent
with the name of the feature and better reflect its design.
I removed the `02-hosts-inventory.e2e` file because it is a duplicate
test of one of the test in the `01-hosts.e2e` file and fixed the error
from that file.
Also, in the inventory Identify test, we test for an element to be not
visible. According to the latest cypress docs, this should be not.exist
instead of not.visible since the cd-modal will not even be present in
the DOM
Fixes: https://tracker.ceph.com/issues/53499 Signed-off-by: Nizamudeen A <nia@redhat.com>
Adds documentation how to change default rbd object size. With the
previous option `--order` it was easy to guess the config name for the
default value, with the current option name `--object-size` thats hard
to guess.
Also extends the documentation for rbd_default_order to include
* how object-size is derived from the configured value
* allowed range of the value
In the first version of this commit I also added min and max for this
parameter (12/25, matching the object size range in `man 8
rbd`/Striping/object-size), but this made some tests fail, since some
seem to set values outside this range (and probably are fine since
included for some time already). To have this a doc-change only, I
removed the range.
Signed-off-by: Mara Sophie Grosch <littlefox@lf-net.org>
Yingxin Cheng [Fri, 24 Dec 2021 08:12:13 +0000 (16:12 +0800)]
crimson/osd: --help-seastar does no longer belong to unknown_args
Now that app_template handles --help-seastar, and prints out all
app-level usages if --help is included in command line options. There is
no need to add a dedicated --help-seastar by ourselves and translate it
to --help.
projects / ceph.git / log