+++ /dev/null
-============
- Operations
-============
-
-.. ===========================
-.. This Document is deprecated
-.. See ceph/doc/cluster-ops
-.. ===========================
-
-.. toctree::
-
- manage/index
- radosgw
+++ /dev/null
-.. _adjusting-crush:
-
-=========================
- Adjusting the CRUSH map
-=========================
-
-There are a few ways to adjust the crush map:
-
-* online, by issuing commands to the monitor
-* offline, by extracting the current map to a file, modifying it, and then reinjecting a new map
-
-For offline changes, some can be made directly with ``crushtool``, and
-others require you to decompile the file to text form, manually edit
-it, and then recompile.
-
-
-Adding a new device (OSD) to the map
-====================================
-
-.. _adjusting-crush-set:
-
-Adding new devices or moving existing devices to new positions in the
-CRUSH hierarchy can be done via the monitor. The general form is::
-
- $ ceph osd crush set <id> <weight> [<loc> [<lo2> ...]]
-
-where
-
- * ``id`` is the numeric device id (the OSD id)
- * ``name`` is an alphanumeric name. By convention Ceph uses
- ``osd.$id``.
- * ``weight`` is a floating point weight value controlling how much
- data the device will be allocated. A decent convention is to make
- this the number of TB the device will store.
- * ``loc`` is a list of ``what=where`` pairs indicating where in the
- CRUSH hierarchy the device will be stored. By default, the
- hierarchy (the ``what``s) includes ``pool`` (the ``default`` pool
- is normally the root of the hierarchy), ``rack``, and ``host``.
- At least one of these location specifiers has to refer to an
- existing point in the hierarchy, and only the lowest (most
- specific) match counts. Beneath that point, any intervening
- branches will be created as needed. Specifying the complete
- location is always sufficient, and also safe in that existing
- branches (and devices) won't be moved around.
-
-For example, if the OSD id is ``123``, we want a weight of ``1.0`` and
-the device is on host ``hostfoo`` and rack ``rackbar``::
-
- $ ceph osd crush set 123 1.0 root=default rack=rackbar host=hostfoo
-
-will add it to the hierarchy, or move it from its previous position.
-(Note: for clusters created prior to v0.52, replace ``root=default``
-with ``pool=default``.) The rack ``rackbar`` and host ``hostfoo``
-will be added as needed, as long as the pool ``default`` exists (as it
-does in the default Ceph CRUSH map generated during cluster creation).
-
-Note that if I later add another device in the same host but specify a
-different pool or rack::
-
- $ ceph osd crush set 124 1.0 root=nondefault rack=weirdrack host=hostfoo
-
-the device will still be placed in host ``hostfoo`` at its current
-location (rack ``rackbar`` and pool ``default``).
-
-
-Moving a bucket to a different position in the hierarchy
-========================================================
-
-To move an existing bucket to a different position in the hierarchy,
-identify the bucket to move by name and specify the new location in
-the same fashion as with ``osd crush set ...``::
-
- $ ceph osd crush move <bucket name> [<loc> [<loc2> ...]]
-
-where
-
- * ``name`` is the name of the bucket to move.
- * ``loc`` is a list of ``what=where`` pairs indicating where the bucket should
- be moved.
-
-
-Adjusting the CRUSH weight
-==========================
-
-You can adjust the CRUSH weight for a device with::
-
- $ ceph osd crush reweight osd.123 2.0
-
-Removing a device
-=================
-
-You can remove a device from the crush map with::
-
- $ ceph osd crush remove osd.123
-
-Tunables
-========
-
-There are several magic numbers that were used in the original CRUSH
-implementation that have proven to be poor choices. To support
-the transition away from them, newer versions of CRUSH (starting with
-the v0.48 argonaut series) allow the values to be adjusted or tuned.
-
-Clusters running recent Ceph releases support using the tunable values
-in the CRUSH maps. However, older clients and daemons will not correctly interact
-with clusters using the "tuned" CRUSH maps. To detect this situation,
-there is now a feature bit ``CRUSH_TUNABLES`` (value 0x40000) to
-reflect support for tunables.
-
-If the OSDMap currently used by the ``ceph-mon`` or ``ceph-osd``
-daemon has non-legacy values, it will require the ``CRUSH_TUNABLES``
-feature bit from clients and daemons who connect to it. This means
-that old clients will not be able to connect.
-
-At some future point in time, newly created clusters will have
-improved default values for the tunables. This is a matter of waiting
-until the support has been present in the Linux kernel clients long
-enough to make this a painless transition for most users.
-
-Impact of legacy values
-~~~~~~~~~~~~~~~~~~~~~~~
-
-The legacy values result in several misbehaviors:
-
- * For hiearchies with a small number of devices in the leaf buckets,
- some PGs map to fewer than the desired number of replicas. This
- commonly happens for hiearchies with "host" nodes with a small
- number (1-3) of OSDs nested beneath each one.
-
- * For large clusters, some small percentages of PGs map to less than
- the desired number of OSDs. This is more prevalent when there are
- several layers of the hierarchy (e.g., row, rack, host, osd).
-
- * When some OSDs are marked out, the data tends to get redistributed
- to nearby OSDs instead of across the entire hierarchy.
-
-Which client versions support tunables
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
- * argonaut series, v0.48.1 or later
- * v0.49 or later
- * Linux kernel version v3.5 or later (for the file system and RBD kernel clients)
-
-A few important points
-~~~~~~~~~~~~~~~~~~~~~~
-
- * Adjusting these values will result in the shift of some PGs between
- storage nodes. If the Ceph cluster is already storing a lot of
- data, be prepared for some fraction of the data to move.
- * The ``ceph-osd`` and ``ceph-mon`` daemons will start requiring the
- ``CRUSH_TUNABLES`` feature of new connections as soon as they get
- the updated map. However, already-connected clients are
- effectively grandfathered in, and will misbehave if they do not
- support the new feature.
- * If the CRUSH tunables are set to non-legacy values and then later
- changed back to the defult values, ``ceph-osd`` daemons will not be
- required to support the feature. However, the OSD peering process
- requires examining and understanding old maps. Therefore, you
- should not run old (pre-v0.48) versions of the ``ceph-osd`` daemon
- if the cluster has previosly used non-legacy CRUSH values, even if
- the latest version of the map has been switched back to using the
- legacy defaults.
-
-Tuning CRUSH
-~~~~~~~~~~~~
-
-If you can ensure that all clients are running recent code, you can
-adjust the tunables by extracting the CRUSH map, modifying the values,
-and reinjecting it into the cluster.
-
-* Extract the latest CRUSH map::
-
- ceph osd getcrushmap -o /tmp/crush
-
-* Adjust tunables. These values appear to offer the best behavior
- for both large and small clusters we tested with. You will need to
- additionally specify the ``--enable-unsafe-tunables`` argument to
- ``crushtool`` for this to work. Please use this option with
- extreme care.::
-
- crushtool -i /tmp/crush --set-choose-local-tries 0 --set-choose-local-fallback-tries 0 --set-choose-total-tries 50 -o /tmp/crush.new
-
-* Reinject modified map::
-
- ceph osd setcrushmap -i /tmp/crush.new
-
-Legacy values
-~~~~~~~~~~~~~
-
-For reference, the legacy values for the CRUSH tunables can be set
-with::
-
- crushtool -i /tmp/crush --set-choose-local-tries 2 --set-choose-local-fallback-tries 5 --set-choose-total-tries 19 -o /tmp/crush.legacy
-
-Again, the special ``--enable-unsafe-tunables`` option is required.
-Further, as noted above, be careful running old versions of the
-``ceph-osd`` daemon after reverting to legacy values as the feature
-bit is not perfectly enforced.
-
+++ /dev/null
-.. _failures-osd:
-
-==========================
- Recovering from failures
-==========================
-
-The current health of the Ceph cluster, as known by the monitors, can
-be checked with the ``ceph health`` command. If all is well, you get::
-
- $ ceph health
- HEALTH_OK
-
-If there are problems, you will see something like::
-
- $ ceph health
- HEALTH_WARN short summary of problem(s)
-
-or::
-
- $ ceph health
- HEALTH_ERROR short summary of very serious problem(s)
-
-To get more detail::
-
- $ ceph health detail
- HEALTH_WARN short description of problem
-
- one problem
- another problem
- yet another problem
- ...
-
-.. toctree::
-
- radosgw
-
+++ /dev/null
-====================================
- Recovering from ``radosgw`` failure
-====================================
-
-
-HTTP Request Errors
-===================
-
-Examining the access and error logs for the web server itself is
-probably the first step in identifying what is going on. If there is
-a 500 error, that usually indicates a problem communicating with the
-``radosgw`` daemon. Ensure the daemon is running, its socket path is
-configured, and that the web server is looking for it in the proper
-location.
-
-
-Crashed ``radosgw`` process
-===========================
-
-If the ``radosgw`` process dies, you will normally see a 500 error
-from the web server (apache, nginx, etc.). In that situation, simply
-restarting radosgw will restore service.
-
-To diagnose the cause of the crash, check the log in ``/var/log/ceph``
-and/or the core file (if one was generated).
-
-
-Blocked ``radosgw`` Requests
-============================
-
-If some (or all) radosgw requests appear to be blocked, you can get
-some insight into the internal state of the ``radosgw`` daemon via
-its admin socket. By default, there will be a socket configured to
-reside in ``/var/run/ceph``, and the daemon can be queried with::
-
- $ ceph --admin-daemon /var/run/ceph/client.rgw help
- help list available commands
- objecter_requests show in-progress osd requests
- perfcounters_dump dump perfcounters value
- perfcounters_schema dump perfcounters schema
- version get protocol version
-
-Of particular interest::
-
- $ ceph --admin-daemon /var/run/ceph/client.rgw objecter_requests
- ...
-
-will dump information about current in-progress requests with the
-RADOS cluster. This allows one to identify if any requests are blocked
-by a non-responsive ceph-osd. For example, one might see::
-
- { "ops": [
- { "tid": 1858,
- "pg": "2.d2041a48",
- "osd": 1,
- "last_sent": "2012-03-08 14:56:37.949872",
- "attempts": 1,
- "object_id": "fatty_25647_object1857",
- "object_locator": "@2",
- "snapid": "head",
- "snap_context": "0=[]",
- "mtime": "2012-03-08 14:56:37.949813",
- "osd_ops": [
- "write 0~4096"]},
- { "tid": 1873,
- "pg": "2.695e9f8e",
- "osd": 1,
- "last_sent": "2012-03-08 14:56:37.970615",
- "attempts": 1,
- "object_id": "fatty_25647_object1872",
- "object_locator": "@2",
- "snapid": "head",
- "snap_context": "0=[]",
- "mtime": "2012-03-08 14:56:37.970555",
- "osd_ops": [
- "write 0~4096"]}],
- "linger_ops": [],
- "pool_ops": [],
- "pool_stat_ops": [],
- "statfs_ops": []}
-
-In this dump, two requests are in progress. The ``last_sent`` field is
-the time the RADOS request was sent. If this is a while ago, it suggests
-that the OSD is not responding. For example, for request 1858, you could
-check the OSD status with::
-
- $ ceph pg map 2.d2041a48
- osdmap e9 pg 2.d2041a48 (2.0) -> up [1,0] acting [1,0]
-
-This tells us to look at ``osd.1``, the primary copy for this PG::
-
- $ ceph --admin-daemon /var/run/ceph/osd.1.asok
- { "num_ops": 651,
- "ops": [
- { "description": "osd_op(client.4124.0:1858 fatty_25647_object1857 [write 0~4096] 2.d2041a48)",
- "received_at": "1331247573.344650",
- "age": "25.606449",
- "flag_point": "waiting for sub ops",
- "client_info": { "client": "client.4124",
- "tid": 1858}},
- ...
-
-The ``flag_point`` field indicates that the OSD is currently waiting
-for replicas to respond, in this case ``osd.0``.
+++ /dev/null
-=========================
- Managing a Ceph cluster
-=========================
-
-.. toctree::
-
- failures/index
- crush
+++ /dev/null
-.. index:: RADOS Gateway, radosgw
-
-=========================================
- Radosgw installation and administration
-=========================================
-
-RADOS Gateway (radosgw or rgw) provides a RESTful API to the object
-store. It interfaces with a web server via FastCGI, and with RADOS via
-libradospp.
-
-Configuring Ceph for RADOS Gateway
-----------------------------------
-
-In order for a host to act as a RADOS gateway, you must add a
-``[client.radosgw.<name>]`` section to your Ceph configuration file
-(typically ``/etc/ceph/ceph.conf``):
-
-::
-
- [client.radosgw.gateway]
- host = gateway
- rgw socket path = /tmp/radosgw.sock
-
-``host`` is the name of the host running radosgw. ``keyring`` points
-to the keyring file for Cephx authentication. ``rgw socket path`` is
-the location of the UNIX socket which radosgw binds to.
-
-If your Ceph cluster has Cephx authentication enabled (highly
-recommended) you also need to add the following option to tell radosgw
-where it finds its authentication key:
-
-::
-
- [client.radosgw.gateway]
- keyring = /etc/ceph/keyring.radosgw.gateway
-
-
-Creating authentication credentials
------------------------------------
-
-To allow radosgw to sucessfully authenticate with the Ceph cluster,
-use the ``ceph-authtool`` command to create a key and set its
-capabilities:
-
-::
-
- ceph-authtool -C -n client.radosgw.gateway \
- --gen-key /etc/ceph/keyring.radosgw.gateway
- ceph-authtool -n client.radosgw.gateway \
- --cap mon 'allow r' --cap osd 'allow rwx' --cap mds 'allow' \
- /etc/ceph/keyring.radosgw.gateway
-
-Finally, add this key to the authentication entries:
-
-::
-
- ceph auth add client.radosgw.gateway \
- --in-file=/etc/ceph/keyring.radosgw.gateway
-
-
-Configuring the web server for radosgw
---------------------------------------
-
-
-The radosgw FastCGI wrapper
----------------------------
-
-A wrapper script, customarily named ``radosgw.cgi`` needs to go into
-your preferred location -- typically your web server root directory.
-
-::
-
- #!/bin/sh
- exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway
-
-
-The ``-c`` option may be omitted if your Ceph configuration file
-resides in its default location ((``/etc/ceph/ceph.conf``)). The
-``-n`` option identifies the ``client`` section in the configuration
-file that radosgw should parse -- if omitted, this would default to
-``client.admin``.
-
-Configuring Apache for radosgw
-------------------------------
-
-The recommended way of deploying radosgw is with Apache and
-``mod_fastcgi``. Ensure that both ``mod_fastcgi`` and ``mod_rewrite``
-are enabled in your Apache configuration. Set the
-``FastCGIExternalServer`` option to point to the radosgw FastCGI
-wrapper.
-
-::
-
- <IfModule mod_fastcgi.c>
- FastCgiExternalServer /var/www/radosgw.fcgi -socket /tmp/radosgw.sock
- </IfModule>
-
-
-Then, create a virtual host configuration as follows:
-
-::
-
- <VirtualHost *:80>
- ServerName radosgw.example.com
- ServerAlias rgw.example.com
- ServerAdmin webmaster@example.com
- DocumentRoot /var/www
-
- <IfModule mod_rewrite.c>
- RewriteEngine On
- RewriteRule ^/(.*) /radosgw.fcgi?%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
- </IfModule>
-
- <IfModule mod_fastcgi.c>
- <Directory /var/www>
- Options +ExecCGI
- AllowOverride All
- SetHandler fastcgi-script
- Order allow,deny
- Allow from all
- AuthBasicAuthoritative Off
- </Directory>
- </IfModule>
-
- AllowEncodedSlashes On
- ServerSignature Off
- </VirtualHost>
-
-
-Starting the daemons
---------------------
-
-For the gateway to become operational, start both the radosgw daemon
-and your web server:
-
-::
-
- service radosgw start
- service apache start
-
-
-Creating users
---------------
-
-In order to be able to use the RESTful API, create a user with the
-``radosgw-admin`` utility:
-
-::
-
- $ radosgw-admin user create --uid=johndoe --display-name="John Doe" --email=john@example.com
- { "user_id": "johndoe",
- "rados_uid": 0,
- "display_name": "John Doe",
- "email": "john@example.com",
- "suspended": 0,
- "subusers": [],
- "keys": [
- { "user": "johndoe",
- "access_key": "QFAMEDSJP5DEKJO0DDXY",
- "secret_key": "iaSFLDVvDdQt6lkNzHyW4fPLZugBAI1g17LO0+87"}],
- "swift_keys": []}
-
-Note that creating a user also creates an ``access_key`` and
-``secret_key`` entry for use with any S3 API-compatible client.
-
-
-Enabling Swift access
----------------------
-
-Allowing access to the object store with Swift (OpenStack Object
-Storage) compatible clients requires an additional step, the creation
-of a subuser and a Swift access key.
-
-::
-
- # radosgw-admin subuser create --uid=johndoe --subuser=johndoe:swift --access=full
- { "user_id": "johndoe",
- "rados_uid": 0,
- "display_name": "John Doe",
- "email": "john@example.com",
- "suspended": 0,
- "subusers": [
- { "id": "johndoe:swift",
- "permissions": "full-control"}],
- "keys": [
- { "user": "johndoe",
- "access_key": "QFAMEDSJP5DEKJO0DDXY",
- "secret_key": "iaSFLDVvDdQt6lkNzHyW4fPLZugBAI1g17LO0+87"}],
- "swift_keys": []}
-
- # radosgw-admin key create --subuser=johndoe:swift --key-type=swift
- { "user_id": "johndoe",
- "rados_uid": 0,
- "display_name": "John Doe",
- "email": "john@example.com",
- "suspended": 0,
- "subusers": [
- { "id": "johndoe:swift",
- "permissions": "full-control"}],
- "keys": [
- { "user": "johndoe",
- "access_key": "QFAMEDSJP5DEKJO0DDXY",
- "secret_key": "iaSFLDVvDdQt6lkNzHyW4fPLZugBAI1g17LO0+87"}],
- "swift_keys": [
- { "user": "johndoe:swift",
- "secret_key": "E9T2rUZNu2gxUjcwUBO8n\/Ev4KX6\/GprEuH4qhu1"}]}
-
-With this configuration, you are able to use any Swift client to
-connect to and use radosgw. As an example, you might use the ``swift``
-command-line client utility that ships with the OpenStack Object
-Storage packages.
-
-::
-
- $ swift -V 1.0 -A http://radosgw.example.com/auth \
- -U johndoe:swift -K E9T2rUZNu2gxUjcwUBO8n\/Ev4KX6\/GprEuH4qhu1 \
- post test
- $ swift -V 1.0 -A http://radosgw.example.com/auth \
- -U johndoe:swift -K E9T2rUZNu2gxUjcwUBO8n\/Ev4KX6\/GprEuH4qhu1 \
- upload test myfile
-
-Note that the radosgw ``user:subuser`` tuple maps to the
-``tenant:user`` tuple expected by Swift.
-
-Note also that the radosgw Swift authentication service only supports
-built-in Swift authentication (``-V 1.0``) at this point. There is
-currently no way to make radosgw authenticate users via OpenStack
-Identity Service (Keystone).
projects / ceph.git / commitdiff