There are two types of user:
-- **User:** The term "user" refers to user of the S3 interface.
+- **User:** The term "user" refers to user of the S3 interface.
- **Subuser:** The term "subuser" refers to a user of the Swift interface. A
subuser is associated with a user.
To create a user (S3 interface), run a command of the following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin user create --uid={username} --display-name="{display-name}" [--email={email}]
For example:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin user create --uid=johndoe --display-name="John Doe" --email=john@example.com
To create a subuser (a user of the Swift interface) for the user, specify the
user ID (``--uid={username}``), a subuser ID, and the subuser's access level:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin subuser create --uid={uid} --subuser={uid} --access=[ read | write | readwrite | full ]
For example:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin subuser create --uid=johndoe --subuser=johndoe:swift --access=full
To get information about a user, specify ``user info`` and the user ID
(``--uid={username}``). Use a command of the following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin user info --uid=johndoe
keys and secrets, email addresses, display names, and access levels. Use a
command of the following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin user modify --uid=johndoe --display-name="John E. Doe"
To modify subuser values, specify ``subuser modify``, user ID and the subuser
ID. Use a command of the following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin subuser modify --uid=johndoe --subuser=johndoe:swift --access=full
user, specify ``user suspend`` and the user ID in a command of the following
form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin user suspend --uid=johndoe
To re-enable a suspended user, provide ``user enable`` and specify the user ID
in a command of the following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin user enable --uid=johndoe
command and provide the user ID of the user to be removed. Use a command of the
following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin user rm --uid=johndoe
To remove the subuser, use the command ``subuser rm`` and provide the subuser
ID of the subuser to be removed. Use a command of the following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin subuser rm --subuser=johndoe:swift
with the UID.
-Add or Remove a Key
---------------------
+Add or Remove a Key
+-------------------
Both users and subusers require a key to access the S3 or Swift interface. To
use S3, the user needs a key pair which is composed of an access key and a
To add a specific S3 key pair for a user, run a command of the following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin key create --uid=foo --key-type=s3 --access-key fooAccessKey --secret-key fooSecretKey
To attach a specific Swift secret key for a subuser, run a command of the
following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin key create --subuser=foo:bar --key-type=swift --secret-key barSecret
.. note:: A subuser can have only one Swift secret key.
-Associating subusers with S3 key pairs
+Associating Subusers With S3 Key Pairs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Subusers can also be used with S3 APIs if the subuser is associated with a S3
key pair. To associate a subuser with an S3 key pair, run a command of the
following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin key create --subuser=foo:bar --key-type=s3 --access-key barAccessKey --secret-key barSecretKey
}
-Removing S3 key pairs
+Removing S3 Key Pairs
~~~~~~~~~~~~~~~~~~~~~
To remove a S3 key pair, specify the access key to be removed. Run a command of the following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin key rm --uid=foo --key-type=s3 --access-key=fooAccessKey
-Removing Swift secret keys
+Removing Swift Secret Keys
~~~~~~~~~~~~~~~~~~~~~~~~~~
To remove a Swift secret key, run a command of the following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin key rm --subuser=foo:bar --key-type=swift
To add administrative capabilities to a user, run a command of the following
form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin caps add --uid={uid} --caps={caps}
-
You can add read, write or all capabilities to users, buckets, metadata and
usage (utilization). To do this, use a command-line option of the following
form:
-.. prompt:: bash
+::
- --caps="[users|buckets|metadata|usage|zone|amz-cache|info|bilog|mdlog|datalog|user-policy|oidc-provider|roles|ratelimit]=[\*|read|write|read, write]"
+ --caps="[users|buckets|metadata|usage|zone|amz-cache|info|bilog|mdlog|datalog|user-policy|oidc-provider|roles|ratelimit|user-info-without-keys|accounts]=[\*|read|write|read, write]"
For example:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin caps add --uid=johndoe --caps="users=*;buckets=*"
To remove administrative capabilities from a user, run a command of the
following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin caps rm --uid=johndoe --caps={caps}
Before you enable a quota, you must first set the quota parameters.
To set quota parameters, run a command of the following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin quota set --quota-scope=user --uid=<uid> [--max-objects=<num objects>] [--max-size=<max size>]
For example:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin quota set --quota-scope=user --uid=johndoe --max-objects=1024 --max-size=1024B
After a user quota is set, it must be enabled in order to take effect. To enable a user quota, run a command of the following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin quota enable --quota-scope=user --uid=<uid>
To disable an enabled user quota, run a command of the following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin quota disable --quota-scope=user --uid=<uid>
independent of the user. To set a bucket quota, run a command of the following
form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin quota set --uid=<uid> --quota-scope=bucket [--max-objects=<num objects>] [--max-size=<max size]
After a bucket quota has been set, it must be enabled in order to take effect.
To enable a bucket quota, run a command of the following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin quota enable --quota-scope=bucket --uid=<uid>
To disable an enabled bucket quota, run a command of the following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin quota disable --quota-scope=bucket --uid=<uid>
API. To read user quota setting information with the CLI interface,
run a command of the following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin user info --uid=<uid>
update quota statistics for all users and all buckets in order to retrieve the
latest quota statistics, run a command of the following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin user stats --uid=<uid> --sync-stats
To see how much of a quota a user has consumed, run a command of the following
form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin user stats --uid=<uid>
You can read and write global quota settings in the period configuration. To
view the global quota settings, run the following command:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin global quota get
counterparts of the ``quota set``, ``quota enable``, and ``quota disable``
commands, as in the following examples:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin global quota set --quota-scope bucket --max-objects 1024
radosgw-admin global quota enable --quota-scope bucket
This means that requests will proceed even if the bucket rate limit or user
rate limit is reached during the execution of the request. The RGW keeps track
of a "debt" consisting of bytes used in excess of the configured value; users
-or buckets that incur this kind of debt are prevented from sending more
+or buckets that incur this kind of debt are prevented from sending more
requests until the "debt" has been repaid. The maximum size of the "debt" is
twice the max-read/write-bytes per minute. If "user A" is subject to a 1-byte
-read limit per minute and they attempt to GET an object that is 1 GB in size,
+read limit per minute and they attempt to ``GET`` an object that is 1 GB in size,
then the ``GET`` action will fail. After "user A" has completed this 1 GB
operation, RGW blocks the user's requests for up to two minutes. After this
time has elapsed, "user A" will be able to send ``GET`` requests again.
parameters. The following is the general form of commands that set rate limit
parameters:
-.. prompt:: bash
+.. prompt:: bash #
- radosgw-admin ratelimit set --ratelimit-scope=user --uid=<uid>
- <[--max-read-ops=<num ops>] [--max-read-bytes=<num bytes>]
- [--max-write-ops=<num ops>] [--max-write-bytes=<num bytes>]>
+ radosgw-admin ratelimit set --ratelimit-scope=user --uid=<uid> \
+ <[--max-read-ops=<num ops>] [--max-read-bytes=<num bytes>] \
+ [--max-write-ops=<num ops>] [--max-write-bytes=<num bytes>]>
An example of using ``radosgw-admin ratelimit set`` to set a rate limit might
look like this:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin ratelimit set --ratelimit-scope=user --uid=johndoe --max-read-ops=1024 --max-write-bytes=10240
-
A value of ``0`` assigned to ``--max-read-ops``, ``--max-read-bytes``,
``--max-write-ops``, or ``--max-write-bytes`` disables the specified rate
limit.
The following is the general form of the command that returns the current
configured limit parameters:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin ratelimit get --ratelimit-scope=user --uid=<uid>
An example of using ``radosgw-admin ratelimit get`` to return the rate limit
parameters might look like this:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin ratelimit get --ratelimit-scope=user --uid=johndoe
After you have set a user rate limit, you must enable it in order for it to
take effect. Run a command of the following form to enable a user rate limit:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin ratelimit enable --ratelimit-scope=user --uid=<uid>
To disable an enabled user rate limit, run a command of the following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin ratelimit disable --ratelimit-scope=user --uid=johndoe
Before you enable a rate limit, you must first set the rate limit parameters.
The following is the general form of commands that set rate limit parameters:
-.. prompt:: bash
+.. prompt:: bash #
- radosgw-admin ratelimit set --ratelimit-scope=bucket --bucket=<bucket> <[--max-read-ops=<num ops>] [--max-read-bytes=<num bytes>]
- [--max-write-ops=<num ops>] [--max-write-bytes=<num bytes>]>
+ radosgw-admin ratelimit set --ratelimit-scope=bucket --bucket=<bucket> \
+ <[--max-read-ops=<num ops>] [--max-read-bytes=<num bytes>] \
+ [--max-write-ops=<num ops>] [--max-write-bytes=<num bytes>]>
An example of using ``radosgw-admin ratelimit set`` to set a rate limit for a
bucket might look like this:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin ratelimit set --ratelimit-scope=bucket --bucket=mybucket --max-read-ops=1024 --max-write-bytes=10240
The following is the general form of the command that returns the current
configured limit parameters:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin ratelimit get --ratelimit-scope=bucket --bucket=<bucket>
An example of using ``radosgw-admin ratelimit get`` to return the rate limit
parameters for a bucket might look like this:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin ratelimit get --ratelimit-scope=bucket --bucket=mybucket
general form of the ``radosgw-admin ratelimit enable`` command that enables
bucket rate limits:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin ratelimit enable --ratelimit-scope=bucket --bucket=<bucket>
An enabled bucket rate limit can be disabled by running a command of the following form:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin ratelimit disable --ratelimit-scope=bucket --uid=mybucket
You can read and write global rate limit settings in the period's configuration.
To view the global rate limit settings, run the following command:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin global ratelimit get
disable`` commands. Per-user and per-bucket ratelimit configurations override
the global configuration:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin global ratelimit set --ratelimit-scope bucket --max-read-ops=1024
radosgw-admin global ratelimit enable --ratelimit-scope bucket
The global rate limit can be used to configure the scope of the rate limit for
all authenticated users:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin global ratelimit set --ratelimit-scope user --max-read-ops=1024
radosgw-admin global ratelimit enable --ratelimit-scope user
The global rate limit can be used to configure the scope of the rate limit for
all unauthenticated users:
-.. prompt:: bash
+.. prompt:: bash #
radosgw-admin global ratelimit set --ratelimit-scope=anonymous --max-read-ops=1024
radosgw-admin global ratelimit enable --ratelimit-scope=anonymous
start date, end date, and whether to show log entries. The following is an example
of such a command:
-.. prompt:: bash $
+.. prompt:: bash #
radosgw-admin usage show --uid=johndoe --start-date=2012-03-01 --end-date=2012-04-01
You can show a summary of usage information for all users by omitting the user
ID, as in the following example command:
-.. prompt:: bash $
+.. prompt:: bash #
radosgw-admin usage show --show-log-entries=false
You can also specify date ranges for trim operations, as in the following
example commands:
-.. prompt:: bash $
+.. prompt:: bash #
radosgw-admin usage trim --start-date=2010-01-01 --end-date=2010-12-31
radosgw-admin usage trim --uid=johndoe
* ``connection`` (container)
-Represents a connection to the remote cloud service. Contains ``connection_id``, ``access_key``,
-``secret``, ``endpoint``, and ``host_style``.
+ Represents a connection to the remote cloud service. Contains ``connection_id``, ``access_key``,
+ ``secret``, ``endpoint``, and ``host_style``.
* ``access_key`` (string)
-The remote cloud access key that will be used for a specific connection.
+ The remote cloud access key that will be used for a specific connection.
* ``secret`` (string)
-The secret key for the remote cloud service.
+ The secret key for the remote cloud service.
* ``endpoint`` (string)
-URL of remote cloud service endpoint.
+ URL of remote cloud service endpoint.
* ``host_style`` (path | virtual)
-Type of host style to be used when accessing remote cloud endpoint (default: ``path``).
+ Type of host style to be used when accessing remote cloud endpoint (default: ``path``).
* ``acls`` (array)
-Contains a list of ``acl_mappings``.
+ Contains a list of ``acl_mappings``.
* ``acl_mapping`` (container)
-Each ``acl_mapping`` structure contains ``type``, ``source_id``, and ``dest_id``. These
-will define the ACL mutation that will be done on each object. An ACL mutation allows converting source
-user id to a destination id.
+ Each ``acl_mapping`` structure contains ``type``, ``source_id``, and ``dest_id``. These
+ will define the ACL mutation that will be done on each object. An ACL mutation allows converting source
+ user id to a destination id.
* ``type`` (id | email | uri)
-ACL type: ``id`` defines user id, ``email`` defines user by email, and ``uri`` defines user by ``uri`` (group).
+ ACL type: ``id`` defines user id, ``email`` defines user by email, and ``uri`` defines user by ``uri`` (group).
* ``source_id`` (string)
-ID of user in the source zone.
+ ID of user in the source zone.
* ``dest_id`` (string)
-ID of user in the destination.
+ ID of user in the destination.
* ``target_path`` (string)
-A string that defines how the target path is created. The target path specifies a prefix to which
-the source object name is appended. The target path configurable can include any of the following
-variables:
+ A string that defines how the target path is created. The target path specifies a prefix to which
+ the source object name is appended. The target path configurable can include any of the following
+ variables:
-* ``sid``: unique string that represents the sync instance ID
-* ``zonegroup``: the zonegroup name
-* ``zonegroup_id``: the zonegroup ID
-* ``zone``: the zone name
-* ``zone_id``: the zone id
-* ``bucket``: source bucket name
-* ``owner``: source bucket owner ID
-
-For example: ``target_path = rgwx-${zone}-${sid}/${owner}/${bucket}``
+ * ``sid``: unique string that represents the sync instance ID
+ * ``zonegroup``: the zonegroup name
+ * ``zonegroup_id``: the zonegroup ID
+ * ``zone``: the zone name
+ * ``zone_id``: the zone id
+ * ``bucket``: source bucket name
+ * ``owner``: source bucket owner ID
+ For example: ``target_path = rgwx-${zone}-${sid}/${owner}/${bucket}``
* ``acl_profiles`` (array)
-An array of ``acl_profile``.
+ An array of ``acl_profile``.
* ``acl_profile`` (container)
-Each profile contains ``acls_id`` (string) that represents the profile, and ``acls`` array that
-holds a list of ``acl_mappings``.
+ Each profile contains ``acls_id`` (string) that represents the profile, and ``acls`` array that
+ holds a list of ``acl_mappings``.
* ``profiles`` (array)
-A list of profiles. Each profile contains the following:
+ A list of profiles. Each profile contains the following:
-* ``source_bucket``: either a bucket name, or a bucket prefix (if ends with ``*``) that defines the source bucket(s) for this profile
-* ``target_path``: as defined above
-* ``connection_id``: ID of the connection that will be used for this profile
-* ``acls_id``: ID of ACLs profile that will be used for this profile
+ * ``source_bucket``: either a bucket name, or a bucket prefix (if ends with ``*``) that defines the source bucket(s) for this profile
+ * ``target_path``: as defined above
+ * ``connection_id``: ID of the connection that will be used for this profile
+ * ``acls_id``: ID of ACLs profile that will be used for this profile
S3 Specific Configurables
* ``multipart_sync_threshold`` (integer)
-Objects this size or larger will be synced to the cloud using multipart upload.
+ Objects this size or larger will be synced to the cloud using multipart upload.
* ``multipart_min_part_size`` (integer)
-Minimum parts size to use when syncing objects using multipart upload.
+ Minimum parts size to use when syncing objects using multipart upload.
How to Configure
Configuration array entries can be accessed by specifying the specific entry to be referenced enclosed
-in square brackets, and adding new array entry can be done by using `[]`. Index value of `-1` references
+in square brackets, and adding new array entry can be done by using ``[]``. Index value of ``-1`` references
the last entry in the array. At the moment it is not possible to create a new entry and reference it
again at the same command.
-For example, creating a new profile for buckets starting with {prefix}:
+For example, creating a new profile for buckets starting with ``{prefix}``:
.. prompt:: bash #
--- /dev/null
+=======
+Metrics
+=======
+
+The Ceph Object Gateway uses :ref:`Perf Counters` to track metrics. The counters can be labeled (:ref:`Labeled Perf Counters`). When counters are labeled, they are stored in the Ceph Object Gateway specific caches.
+
+These metrics can be sent to the time series database Prometheus to visualize a cluster wide view of usage data (ex: number of S3 put operations on a specific bucket) over time.
+
+.. contents::
+
+Op Metrics
+==========
+
+The following metrics related to S3 or Swift operations are tracked per Ceph Object Gateway.
+
+.. list-table:: Radosgw Op Metrics
+ :widths: 25 25 75
+ :header-rows: 1
+
+ * - Name
+ - Type
+ - Description
+ * - put_obj_ops
+ - Counter
+ - Number of put operations
+ * - put_obj_bytes
+ - Counter
+ - Number of bytes put
+ * - put_obj_lat
+ - Gauge
+ - Total latency of put operations
+ * - get_obj_ops
+ - Counter
+ - Number of get operations
+ * - get_obj_bytes
+ - Counter
+ - Number of bytes from get requests
+ * - get_obj_lat
+ - Gauge
+ - Total latency of get operations
+ * - del_obj_ops
+ - Counter
+ - Number of delete object operations
+ * - del_obj_bytes
+ - Counter
+ - Number of bytes deleted
+ * - del_obj_lat
+ - Gauge
+ - Total latency of delete object operations
+ * - del_bucket_ops
+ - Counter
+ - Number of delete bucket operations
+ * - del_bucket_lat
+ - Gauge
+ - Total latency of delete bucket operations
+ * - copy_obj_ops
+ - Counter
+ - Number of copy object operations
+ * - copy_obj_bytes
+ - Counter
+ - Number of bytes copied
+ * - copy_obj_lat
+ - Gauge
+ - Total latency of copy object operations
+ * - list_object_ops
+ - Counter
+ - Number of list object operations
+ * - list_object_lat
+ - Gauge
+ - Total latency of list object operations
+ * - list_bucket_ops
+ - Counter
+ - Number of list bucket operations
+ * - list_bucket_lat
+ - Gauge
+ - Total latency of list bucket operations
+
+There are three different sections in the output of the ``counter dump`` and ``counter schema`` commands that show the op metrics and their information.
+The sections are ``rgw_op``, ``rgw_op_per_user``, and ``rgw_op_per_bucket``.
+
+The counters in the ``rgw_op`` section reflect the totals of each op metric for a given Ceph Object Gateway.
+The counters in the ``rgw_op_per_user`` and ``rgw_op_per_bucket`` sections are labeled counters of op metrics for a user or bucket respectively.
+
+Information about op metrics can be seen in the ``rgw_op`` sections of the output of the ``counter schema`` command.
+
+To view op metrics in the Ceph Object Gateway go to the ``rgw_op`` sections of the output of the ``counter dump`` command::
+
+ "rgw_op": [
+ {
+ "labels": {},
+ "counters": {
+ "put_obj_ops": 2,
+ "put_obj_bytes": 5327,
+ "put_obj_lat": {
+ "avgcount": 2,
+ "sum": 2.818064835,
+ "avgtime": 1.409032417
+ },
+ "get_obj_ops": 5,
+ "get_obj_bytes": 5325,
+ "get_obj_lat": {
+ "avgcount": 2,
+ "sum": 0.003000069,
+ "avgtime": 0.001500034
+ },
+ ...
+ "list_buckets_ops": 1,
+ "list_buckets_lat": {
+ "avgcount": 1,
+ "sum": 0.002300000,
+ "avgtime": 0.002300000
+ }
+ }
+ },
+ ]
+
+Op Metrics Labels
+-----------------
+
+Op metrics can also be tracked per-user or per-bucket. These metrics are exported to Prometheus with labels like Bucket = {name} or User = {userid}::
+
+ "rgw_op_per_bucket": [
+ ...
+ {
+ "labels": {
+ "Bucket": "bucket1"
+ },
+ "counters": {
+ "put_obj_ops": 2,
+ "put_obj_bytes": 5327,
+ "put_obj_lat": {
+ "avgcount": 2,
+ "sum": 2.818064835,
+ "avgtime": 1.409032417
+ },
+ "get_obj_ops": 5,
+ "get_obj_bytes": 5325,
+ "get_obj_lat": {
+ "avgcount": 2,
+ "sum": 0.003000069,
+ "avgtime": 0.001500034
+ },
+ ...
+ "list_buckets_ops": 1,
+ "list_buckets_lat": {
+ "avgcount": 1,
+ "sum": 0.002300000,
+ "avgtime": 0.002300000
+ }
+ }
+ },
+ ...
+ ]
+
+:ref:`rgw-multitenancy` allows to use buckets and users of the same name simultaneously. If a user or bucket lies under a tenant, a label for tenant in the form Tenant = {tenantid} is added to the metric.
+
+In a large system with many users and buckets, it may not be tractable to export all metrics to Prometheus. For that reason, the collection of these labeled metrics is disabled by default.
+
+Once enabled, the working set of tracked users and buckets is constrained to limit memory and database usage. As a result, the collection of these labeled metrics will not always be reliable.
+
+
+User & Bucket Counter Caches
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To track op metrics by user the Ceph Object Gateway the config value ``rgw_user_counters_cache`` must be set to ``true``.
+
+To track op metrics by bucket the Ceph Object Gateway the config value ``rgw_bucket_counters_cache`` must be set to ``true``.
+
+These config values are set in Ceph via the command ``ceph config set client.rgw rgw_{user,bucket}_counters_cache true``
+
+Since the op metrics are labeled perf counters, they live in memory. If the Ceph Object Gateway is restarted or crashes, all counters in the Ceph Object Gateway, whether in a cache or not, are lost.
+
+User & Bucket Counter Cache Size & Eviction
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Both ``rgw_user_counters_cache_size`` and ``rgw_bucket_counters_cache_size`` can be used to set number of entries in each cache.
+
+Counters are evicted from a cache once the number of counters in the cache are greater than the cache size config variable. The counters that are evicted are the least recently used (LRU).
+
+For example if the number of buckets exceeded ``rgw_bucket_counters_cache_size`` by 1 and the counters with label ``bucket1`` were the last to be updated, the counters for ``bucket1`` would be evicted from the cache. If S3 operations tracked by the op metrics were done on ``bucket1`` after eviction, all of the metrics in the cache for ``bucket1`` would start at 0.
+
+Cache sizing can depend on a number of factors. These factors include:
+
+#. Number of users in the cluster
+#. Number of buckets in the cluster
+#. Memory usage of the Ceph Object Gateway
+#. Disk and memory usage of Promtheus.
+
+To help calculate the Ceph Object Gateway's memory usage of a cache, it should be noted that each cache entry, encompassing all of the op metrics, is 1360 bytes. This is an estimate and subject to change if metrics are added or removed from the op metrics list.
+
+Sending Metrics to Prometheus
+=============================
+
+To get metrics from a Ceph Object Gateway into the time series database Prometheus, the ceph-exporter daemon must be running and configured to scrape the Radogw's admin socket.
+
+The ceph-exporter daemon scrapes the Ceph Object Gateway's admin socket at a regular interval, defined by the config variable ``exporter_stats_period``.
+
+Prometheus has a configurable interval in which it scrapes the exporter (see: https://prometheus.io/docs/prometheus/latest/configuration/configuration/).
+
+Config Reference
+================
+The following rgw op metrics related settings can be set via ``ceph config set client.rgw CONFIG_VARIABLE VALUE``.
+
+.. confval:: rgw_user_counters_cache
+.. confval:: rgw_user_counters_cache_size
+.. confval:: rgw_bucket_counters_cache
+.. confval:: rgw_bucket_counters_cache_size
+
+The following are notable ceph-exporter related settings can be set via ``ceph config set global CONFIG_VARIABLE VALUE``.
+
+.. confval:: exporter_stats_period
Multisite bucket-granularity sync policy provides fine grained control of data movement between buckets in different zones. It extends the zone sync mechanism. Previously buckets were being treated symmetrically, that is -- each (data) zone holds a mirror of that bucket that should be the same as all the other zones. Whereas leveraging the bucket-granularity sync policy is possible for buckets to diverge, and a bucket can pull data from other buckets (ones that don't share its name or its ID) in different zone. The sync process was assuming therefore that the bucket sync source and the bucket sync destination were always referring to the same bucket, now that is not the case anymore.
-The sync policy supersedes the old zonegroup coarse configuration (sync_from*). The sync policy can be configured at the zonegroup level (and if it is configured it replaces the old style config), but it can also be configured at the bucket level.
+The sync policy supersedes the old zonegroup coarse configuration (``sync_from*``). The sync policy can be configured at the zonegroup level (and if it is configured it replaces the old style config), but it can also be configured at the bucket level.
In the sync policy multiple groups that can contain lists of data-flow configurations can be defined, as well as lists of pipe configurations. The data-flow defines the flow of data between the different zones. It can define symmetrical data flow, in which multiple zones sync data from each other, and it can define directional data flow, in which the data moves in one way from one zone to another.
.. important:: Any changes to the zonegroup policy needs to be applied on the
zonegroup master zone, and require period update and commit. Changes
to the bucket policy needs to be applied on the zonegroup master
- zone. The changes are dynamically handled by rgw.
+ zone. The changes are dynamically handled by RGW.
S3 Replication API
~~~~~~~~~~~~~~~~~~
-The S3 bucket replication api has also been implemented, and allows users to create replication rules between different buckets. Note though that while the AWS replication feature allows bucket replication within the same zone, rgw does not allow it at the moment. However, the rgw api also added a new 'Zone' array that allows users to select to what zones the specific bucket will be synced.
+The S3 bucket replication API has also been implemented, and allows users to create replication rules between different buckets. Note though that while the AWS replication feature allows bucket replication within the same zone, RGW does not allow it at the moment. However, the RGW API also added a new 'Zone' array that allows users to select to what zones the specific bucket will be synced.
Sync Policy Control Reference
Create Sync Flow
~~~~~~~~~~~~~~~~
-- To create or update directional sync flow:
+To create or update directional sync flow:
.. prompt:: bash #
--dest-zone=<dest_zone>
-- To create or update symmetrical sync flow:
+To create or update symmetrical sync flow:
.. prompt:: bash #
Remove Sync Flow Zones
~~~~~~~~~~~~~~~~~~~~~~
-- To remove directional sync flow:
+To remove directional sync flow:
.. prompt:: bash #
--dest-zone=<dest_zone>
-- To remove specific zones from symmetrical sync flow:
+To remove specific zones from symmetrical sync flow:
.. prompt:: bash #
Where zones are a comma separated list of all zones to remove from the flow.
-- To remove symmetrical sync flow:
+To remove symmetrical sync flow:
.. prompt:: bash #
Example 1: Two Zones, Complete Mirror
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-This is similar to older (pre ``Octopus``) sync capabilities, but being done via the new sync policy engine. Note that changes to the zonegroup sync policy require a period update and commit.
+This is similar to older (pre Octopus) sync capabilities, but being done via the new sync policy engine. Note that changes to the zonegroup sync policy require a period update and commit.
.. prompt:: bash [us-east]#
radosgw-admin period update --commit
-Note that us-west has two dests:
+Note that ``us-west`` has two destinations:
.. prompt:: bash [us-west]#
}
-Whereas us-west-2 has only source and no destinations:
+Whereas ``us-west-2`` has only source and no destinations:
.. prompt:: bash [us-west-2]#
can also limit it to specific zones, for example the following will
-only sync data originated in us-west:
+only sync data originated in ``us-west``:
.. prompt:: bash [us-east]#
Example 6: Sync to Different Bucket
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The same mechanism can work for configuring data to be synced to (vs. synced from as in the previous example). Note that internally data is still pulled from the source at the destination zone:
+The same mechanism can work for configuring data to be synced to (vs. synced from as in the previous example). Note that internally data is still pulled from the source at the destination zone:
Set ``buck6`` to "push" data to ``buck5``:
A single-zone configuration typically consists of two things:
#. One "zonegroup", which contains one zone.
-#. One or more `ceph-radosgw` instances that have `ceph-radosgw` client
+#. One or more ``ceph-radosgw`` instances that have ``ceph-radosgw`` client
requests load-balanced between them.
-In a typical single-zone configuration, there are multiple `ceph-radosgw`
+In a typical single-zone configuration, there are multiple ``ceph-radosgw``
instances that make use of a single Ceph storage cluster.
Varieties of Multi-site Configuration
- **Multi-zone:** The "multi-zone" configuration has a complex topology. A
multi-zone configuration consists of one zonegroup and multiple zones. Each
- zone consists of one or more `ceph-radosgw` instances. **Each zone is backed
+ zone consists of one or more ``ceph-radosgw`` instances. **Each zone is backed
by its own Ceph Storage Cluster.**
The presence of multiple zones in a given zonegroup provides disaster
The application on the right is both writing and reading data from the Ceph
Cluster, by means of the RADOS Gateway (RGW). The application on the left is
only *reading* data from the Ceph Cluster, by means of an instance of RADOS
-Gateway (RGW). In both cases (read-and-write and read-only), the transmssion of
+Gateway. In both cases (read-and-write and read-only), the transmssion of
data is handled RESTfully.
In the middle of this diagram, we see two zones, each of which contains an
-instance of RADOS Gateway (RGW). These instances of RGW are handling the
+instance of RADOS Gateway. These instances of RADOS Gateway are handling the
movement of data from the applications to the zonegroup. The arrow from the
master zone (US-EAST) to the secondary zone (US-WEST) represents an act of data
synchronization.
.. prompt:: bash #
- radosgw-admin realm pull --url={url-to-master-zone-gateway}
- --access-key={access-key} --secret={secret}
+ radosgw-admin realm pull --url={url-to-master-zone-gateway} \
+ --access-key={access-key} --secret={secret}
.. note:: Pulling the realm configuration also retrieves the remote's current
period configuration, and makes it the current period on this host as well.
.. prompt:: bash #
radosgw-admin zone create --rgw-zonegroup={zone-group-name} \
- --rgw-zone={zone-name} \
- --access-key={system-key} --secret={secret} \
- --endpoints=http://{fqdn}:80 \
- [--read-only]
+ --rgw-zone={zone-name} \
+ --access-key={system-key} --secret={secret} \
+ --endpoints=http://{fqdn}:80 \
+ [--read-only]
For example:
.. prompt:: bash #
radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-west \
- --access-key={system-key} --secret={secret} \
- --endpoints=http://rgw2:80
+ --access-key={system-key} --secret={secret} \
+ --endpoints=http://rgw2:80
.. important:: The following steps assume a multi-site configuration that uses
newly installed systems that have not yet begun storing data. **DO NOT
By default, after the successful synchronization of an object there is no
subsequent verification of the object. However, you can enable verification by
setting :confval:`rgw_sync_obj_etag_verify` to ``true``. After this value is
-set to true, an MD5 checksum is used to verify the integrity of the data that
+set to ``true``, an MD5 checksum is used to verify the integrity of the data that
was transferred from the source to the destination. This ensures the integrity
of any object that has been fetched from a remote server over HTTP (including
-multisite sync). This option may decrease the performance of your RGW because
+multi-site sync). This option may decrease the performance of your RGW because
it requires more computation.
.. prompt:: bash #
radosgw-admin zone modify --rgw-zone={zone-name} --master --default \
- --read-only=false
+ --read-only=false
#. Update the period to make the changes take effect.
.. prompt:: bash #
radosgw-admin realm pull --url={url-to-master-zone-gateway} \
- --access-key={access-key} --secret={secret}
+ --access-key={access-key} --secret={secret}
#. Make the recovered zone the master and default zone:
.. prompt:: bash #
radosgw-admin zone modify --rgw-realm=<name> --rgw-zonegroup=<name> \
- --rgw-zone=<name> --endpoints http://<fqdn>:80 \
- --access-key=<access-key> --secret=<secret-key> \
- --master --default
+ --rgw-zone=<name> --endpoints http://<fqdn>:80 \
+
--access-key=<access-key> --secret=<secret-key> \
+ --master --default
6. Create a system user. Replace ``<user-id>`` with the username. Replace
``<display-name>`` with a display name. The display name is allowed to
.. prompt:: bash #
radosgw-admin user create --uid=<user-id> \
- --display-name="<display-name>" \
- --access-key=<access-key> \
- --secret=<secret-key> --system
+ --display-name="<display-name>" \
+ --access-key=<access-key> \
+ --secret=<secret-key> --system
7. Commit the updated configuration:
~~~~~~~~~~~
To set a realm, run ``realm set``, specify the realm name, and use the
-``--infile=`` option (make sure that the ``--infile`` option has an input file
+``--infile=`` option (make sure that the ``--infile`` option has an input file
name as an argument):
.. prompt:: bash #
7. ``zones``: A list of all zones within the zonegroup. Each zone has a name
(required), a list of endpoints (optional), and a setting that determines
- whether the gateway will log metadata and data operations (false by
+ whether the gateway will log metadata and data operations (``false`` by
default).
8. ``placement_targets``: A list of placement targets (optional). Each
the user info).
9. ``default_placement``: The default placement target for the object index and
- object data. Set to ``default-placement`` by default. It is also possible
+ object data. Set to ``default-placement`` by default. It is also possible
to set a per-user default placement in the user info for each user.
Setting a Zonegroup - Procedure
-----
A zone defines a logical group that consists of one or more Ceph Object Gateway
-instances. All RGWs in a given zone serve S3 objects that are backed by RADOS objects that are stored in the same set of pools in the same cluster. Ceph Object Gateway supports zones.
+instances. All Ceph Object Gateways in a given zone serve S3 objects that are backed by RADOS objects that are stored in the same set of pools in the same cluster. Ceph Object Gateway supports zones.
The procedure for configuring zones differs from typical configuration
procedures, because not all of the settings end up in a Ceph configuration
.. prompt:: bash #
radosgw-admin zone create --rgw-zone=<name> \
- [--zonegroup=<zonegroup-name]\
- [--endpoints=<endpoint>[,<endpoint>] \
- [--master] [--default] \
- --access-key $SYSTEM_ACCESS_KEY --secret $SYSTEM_SECRET_KEY
+ [--zonegroup=<zonegroup-name] \
+ [--endpoints=<endpoint>[,<endpoint>] \
+ [--master] [--default] \
+ --access-key $SYSTEM_ACCESS_KEY \
+ --secret $SYSTEM_SECRET_KEY
After you have created the zone, update the period:
.. prompt:: bash #
- radosgw-admin zonegroup remove --zonegroup=<name>\
- --zone=<name>
+ radosgw-admin zonegroup remove --zonegroup=<name> \
+ --zone=<name>
Then, update the period:
----------------------------
When configuring a default zonegroup and zone, the pool name includes
-the zone name. For example:
+the zone name. For example::
-- ``default.rgw.control``
+ default.rgw.control
To change the defaults, include the following settings in your Ceph
configuration file under each ``[client.radosgw.{instance-name}]``
projects / ceph.git / commitdiff