From 818ed4f2025d46a0822d025c5cae5eeb161099c3 Mon Sep 17 00:00:00 2001 From: Sebastian Wagner Date: Thu, 18 Feb 2021 15:17:07 +0100 Subject: [PATCH] doc/cephadm: group RGW mgmt sections into one chapter Signed-off-by: Sebastian Wagner --- doc/cephadm/index.rst | 1 + doc/cephadm/install.rst | 49 +-------- doc/cephadm/rgw.rst | 222 +++++++++++++++++++++++++++++++++++++++ doc/mgr/orchestrator.rst | 167 ----------------------------- 4 files changed, 224 insertions(+), 215 deletions(-) create mode 100644 doc/cephadm/rgw.rst diff --git a/doc/cephadm/index.rst b/doc/cephadm/index.rst index a500e9a7ca4..d1649447890 100644 --- a/doc/cephadm/index.rst +++ b/doc/cephadm/index.rst @@ -32,6 +32,7 @@ versions of Ceph. adoption host-management osd + rgw service-management upgrade Cephadm operations diff --git a/doc/cephadm/install.rst b/doc/cephadm/install.rst index d950ac1cccd..aa2b26e9401 100644 --- a/doc/cephadm/install.rst +++ b/doc/cephadm/install.rst @@ -407,55 +407,8 @@ For example: See :ref:`orchestrator-cli-stateless-services` for manually deploying MDS daemons. -Deploy RGWs -=========== -Cephadm deploys radosgw as a collection of daemons that manage a -particular *realm* and *zone*. (For more information about realms and -zones, see :ref:`multisite`.) - -Note that with cephadm, radosgw daemons are configured via the monitor -configuration database instead of via a `ceph.conf` or the command line. If -that configuration isn't already in place (usually in the -``client.rgw..`` section), then the radosgw -daemons will start up with default settings (e.g., binding to port -80). - -To deploy a set of radosgw daemons for a particular realm and zone, run the -following command: - -.. prompt:: bash # - - ceph orch apply rgw ** ** --placement="** [** ...]" - -For example, to deploy 2 rgw daemons serving the *myorg* realm and the *us-east-1* zone on *myhost1* and *myhost2*: - -.. prompt:: bash # - - ceph orch apply rgw myorg us-east-1 --placement="2 myhost1 myhost2" - -Cephadm will wait for a healthy cluster and automatically create the supplied realm and zone if they do not exist before deploying the rgw daemon(s) - -Alternatively, the realm, zonegroup, and zone can be manually created using ``radosgw-admin`` commands: - -.. prompt:: bash # - - radosgw-admin realm create --rgw-realm= --default - -.. prompt:: bash # - - radosgw-admin zonegroup create --rgw-zonegroup= --master --default - -.. prompt:: bash # - - radosgw-admin zone create --rgw-zonegroup= --rgw-zone= --master --default - -.. prompt:: bash # - - radosgw-admin period update --rgw-realm= --commit - -See :ref:`orchestrator-cli-placement-spec` for details of the placement -specification. +To use the *Ceph Object Gateway*, follow :ref:`cephadm-deploy-rgw`. .. _deploy-cephadm-nfs-ganesha: diff --git a/doc/cephadm/rgw.rst b/doc/cephadm/rgw.rst new file mode 100644 index 00000000000..3964c987a1f --- /dev/null +++ b/doc/cephadm/rgw.rst @@ -0,0 +1,222 @@ +=========== +RGW Service +=========== + +.. _cephadm-deploy-rgw: + +Deploy RGWs +=========== + +Cephadm deploys radosgw as a collection of daemons that manage a +particular *realm* and *zone*. (For more information about realms and +zones, see :ref:`multisite`.) + +Note that with cephadm, radosgw daemons are configured via the monitor +configuration database instead of via a `ceph.conf` or the command line. If +that configuration isn't already in place (usually in the +``client.rgw..`` section), then the radosgw +daemons will start up with default settings (e.g., binding to port +80). + +To deploy a set of radosgw daemons for a particular realm and zone, run the +following command: + +.. prompt:: bash # + + ceph orch apply rgw ** ** --placement="** [** ...]" + +For example, to deploy 2 rgw daemons serving the *myorg* realm and the *us-east-1* zone on *myhost1* and *myhost2*: + +.. prompt:: bash # + + ceph orch apply rgw myorg us-east-1 --placement="2 myhost1 myhost2" + +Cephadm will wait for a healthy cluster and automatically create the supplied realm and zone if they do not exist before deploying the rgw daemon(s) + +Alternatively, the realm, zonegroup, and zone can be manually created using ``radosgw-admin`` commands: + +.. prompt:: bash # + + radosgw-admin realm create --rgw-realm= --default + +.. prompt:: bash # + + radosgw-admin zonegroup create --rgw-zonegroup= --master --default + +.. prompt:: bash # + + radosgw-admin zone create --rgw-zonegroup= --rgw-zone= --master --default + +.. prompt:: bash # + + radosgw-admin period update --rgw-realm= --commit + +See :ref:`orchestrator-cli-placement-spec` for details of the placement +specification. + + +.. _orchestrator-haproxy-service-spec: + +High availability service for RGW +================================= + +This service allows the user to create a high avalilability RGW service +providing a mimimun set of configuration options. + +The orchestrator will deploy and configure automatically several HAProxy and +Keepalived containers to assure the continuity of the RGW service while the +Ceph cluster will have at least 1 RGW daemon running. + +The next image explains graphically how this service works: + +.. image:: ../images/HAProxy_for_RGW.svg + +There are N hosts where the HA RGW service is deployed. This means that we have +an HAProxy and a keeplived daemon running in each of this hosts. +Keepalived is used to provide a "virtual IP" binded to the hosts. All RGW +clients use this "virtual IP" to connect with the RGW Service. + +Each keeplived daemon is checking each few seconds what is the status of the +HAProxy daemon running in the same host. Also it is aware that the "master" keepalived +daemon will be running without problems. + +If the "master" keepalived daemon or the Active HAproxy is not responding, one +of the keeplived daemons running in backup mode will be elected as master, and +the "virtual ip" will be moved to that node. + +The active HAProxy also acts like a load balancer, distributing all RGW requests +between all the RGW daemons available. + +**Prerequisites:** + +* At least two RGW daemons running in the Ceph cluster +* Operating system prerequisites: + In order for the Keepalived service to forward network packets properly to the + real servers, each router node must have IP forwarding turned on in the kernel. + So it will be needed to set this system option:: + + net.ipv4.ip_forward = 1 + + Load balancing in HAProxy and Keepalived at the same time also requires the + ability to bind to an IP address that are nonlocal, meaning that it is not + assigned to a device on the local system. This allows a running load balancer + instance to bind to an IP that is not local for failover. + So it will be needed to set this system option:: + + net.ipv4.ip_nonlocal_bind = 1 + + Be sure to set properly these two options in the file ``/etc/sysctl.conf`` in + order to persist this values even if the hosts are restarted. + These configuration changes must be applied in all the hosts where the HAProxy for + RGW service is going to be deployed. + + +**Deploy of the high availability service for RGW** + +Use the command:: + + ceph orch apply -i + +**Service specification file:** + +It is a yaml format file with the following properties: + +.. code-block:: yaml + + service_type: ha-rgw + service_id: haproxy_for_rgw + placement: + hosts: + - host1 + - host2 + - host3 + spec: + virtual_ip_interface: # ex: eth0 + virtual_ip_address: / # ex: 192.168.20.1/24 + frontend_port: # ex: 8080 + ha_proxy_port: # ex: 1967 + ha_proxy_stats_enabled: # ex: true + ha_proxy_stats_user: # ex: admin + ha_proxy_stats_password: # ex: true + ha_proxy_enable_prometheus_exporter: # ex: true + ha_proxy_monitor_uri: # ex: /haproxy_health + keepalived_user: # ex: admin + keepalived_password: # ex: admin + ha_proxy_frontend_ssl_certificate: ex: + [ + "-----BEGIN CERTIFICATE-----", + "MIIDZTCCAk2gAwIBAgIUClb9dnseOsgJWAfhPQvrZw2MP2kwDQYJKoZIhvcNAQEL", + .... + "-----END CERTIFICATE-----", + "-----BEGIN PRIVATE KEY-----", + .... + "sCHaZTUevxb4h6dCEk1XdPr2O2GdjV0uQ++9bKahAy357ELT3zPE8yYqw7aUCyBO", + "aW5DSCo8DgfNOgycVL/rqcrc", + "-----END PRIVATE KEY-----" + ] + ha_proxy_frontend_ssl_port: # ex: 8090 + ha_proxy_ssl_dh_param: # ex: 1024 + ha_proxy_ssl_ciphers: # ex: ECDH+AESGCM:!MD5 + ha_proxy_ssl_options: # ex: no-sslv3 + haproxy_container_image: # ex: haproxy:2.4-dev3-alpine + keepalived_container_image: # ex: arcts/keepalived:1.2.2 + +where the properties of this service specification are: + +* ``service_type`` + Mandatory and set to "ha-rgw" +* ``service_id`` + The name of the service. +* ``placement hosts`` + The hosts where it is desired to run the HA daemons. An HAProxy and a + Keepalived containers will be deployed in these hosts. + The RGW daemons can run in other different hosts or not. +* ``virtual_ip_interface`` + The physical network interface where the virtual ip will be binded +* ``virtual_ip_address`` + The virtual IP ( and network ) where the HA RGW service will be available. + All your RGW clients must point to this IP in order to use the HA RGW + service . +* ``frontend_port`` + The port used to access the HA RGW service +* ``ha_proxy_port`` + The port used by HAProxy containers +* ``ha_proxy_stats_enabled`` + If it is desired to enable the statistics URL in HAProxy daemons +* ``ha_proxy_stats_user`` + User needed to access the HAProxy statistics URL +* ``ha_proxy_stats_password`` + The password needed to access the HAProxy statistics URL +* ``ha_proxy_enable_prometheus_exporter`` + If it is desired to enable the Promethes exporter in HAProxy. This will + allow to consume RGW Service metrics from Grafana. +* ``ha_proxy_monitor_uri``: + To set the API endpoint where the health of HAProxy daemon is provided +* ``keepalived_user`` + User needed to access keepalived daemons +* ``keepalived_password``: + The password needed to access keepalived daemons +* ``ha_proxy_frontend_ssl_certificate``: + SSl certificate. You must paste the content of your .pem file +* ``ha_proxy_frontend_ssl_port``: + The https port used by HAProxy containers +* ``ha_proxy_ssl_dh_param``: + Value used for the `tune.ssl.default-dh-param` setting in the HAProxy + config file +* ``ha_proxy_ssl_ciphers``: + Value used for the `ssl-default-bind-ciphers` setting in HAProxy config + file. +* ``ha_proxy_ssl_options``: + Value used for the `ssl-default-bind-options` setting in HAProxy config + file. +* ``haproxy_container_image``: + HAProxy image location used to pull the image +* ``keepalived_container_image``: + Keepalived image location used to pull the image + +**Useful hints for the RGW Service:** + +* Good to have at least 3 RGW daemons +* Use at least 3 hosts for the HAProxy for RGW service +* In each host an HAProxy and a Keepalived daemon will be deployed. These + daemons can be managed as systemd services diff --git a/doc/mgr/orchestrator.rst b/doc/mgr/orchestrator.rst index b527a5cef4e..8f228320715 100644 --- a/doc/mgr/orchestrator.rst +++ b/doc/mgr/orchestrator.rst @@ -184,173 +184,6 @@ Service Commands:: ceph orch -.. _orchestrator-haproxy-service-spec: - -High availability service for RGW -================================= - -This service allows the user to create a high avalilability RGW service -providing a mimimun set of configuration options. - -The orchestrator will deploy and configure automatically several HAProxy and -Keepalived containers to assure the continuity of the RGW service while the -Ceph cluster will have at least 1 RGW daemon running. - -The next image explains graphically how this service works: - -.. image:: ../images/HAProxy_for_RGW.svg - -There are N hosts where the HA RGW service is deployed. This means that we have -an HAProxy and a keeplived daemon running in each of this hosts. -Keepalived is used to provide a "virtual IP" binded to the hosts. All RGW -clients use this "virtual IP" to connect with the RGW Service. - -Each keeplived daemon is checking each few seconds what is the status of the -HAProxy daemon running in the same host. Also it is aware that the "master" keepalived -daemon will be running without problems. - -If the "master" keepalived daemon or the Active HAproxy is not responding, one -of the keeplived daemons running in backup mode will be elected as master, and -the "virtual ip" will be moved to that node. - -The active HAProxy also acts like a load balancer, distributing all RGW requests -between all the RGW daemons available. - -**Prerequisites:** - -* At least two RGW daemons running in the Ceph cluster -* Operating system prerequisites: - In order for the Keepalived service to forward network packets properly to the - real servers, each router node must have IP forwarding turned on in the kernel. - So it will be needed to set this system option:: - - net.ipv4.ip_forward = 1 - - Load balancing in HAProxy and Keepalived at the same time also requires the - ability to bind to an IP address that are nonlocal, meaning that it is not - assigned to a device on the local system. This allows a running load balancer - instance to bind to an IP that is not local for failover. - So it will be needed to set this system option:: - - net.ipv4.ip_nonlocal_bind = 1 - - Be sure to set properly these two options in the file ``/etc/sysctl.conf`` in - order to persist this values even if the hosts are restarted. - These configuration changes must be applied in all the hosts where the HAProxy for - RGW service is going to be deployed. - - -**Deploy of the high availability service for RGW** - -Use the command:: - - ceph orch apply -i - -**Service specification file:** - -It is a yaml format file with the following properties: - -.. code-block:: yaml - - service_type: ha-rgw - service_id: haproxy_for_rgw - placement: - hosts: - - host1 - - host2 - - host3 - spec: - virtual_ip_interface: # ex: eth0 - virtual_ip_address: / # ex: 192.168.20.1/24 - frontend_port: # ex: 8080 - ha_proxy_port: # ex: 1967 - ha_proxy_stats_enabled: # ex: true - ha_proxy_stats_user: # ex: admin - ha_proxy_stats_password: # ex: true - ha_proxy_enable_prometheus_exporter: # ex: true - ha_proxy_monitor_uri: # ex: /haproxy_health - keepalived_user: # ex: admin - keepalived_password: # ex: admin - ha_proxy_frontend_ssl_certificate: ex: - [ - "-----BEGIN CERTIFICATE-----", - "MIIDZTCCAk2gAwIBAgIUClb9dnseOsgJWAfhPQvrZw2MP2kwDQYJKoZIhvcNAQEL", - .... - "-----END CERTIFICATE-----", - "-----BEGIN PRIVATE KEY-----", - .... - "sCHaZTUevxb4h6dCEk1XdPr2O2GdjV0uQ++9bKahAy357ELT3zPE8yYqw7aUCyBO", - "aW5DSCo8DgfNOgycVL/rqcrc", - "-----END PRIVATE KEY-----" - ] - ha_proxy_frontend_ssl_port: # ex: 8090 - ha_proxy_ssl_dh_param: # ex: 1024 - ha_proxy_ssl_ciphers: # ex: ECDH+AESGCM:!MD5 - ha_proxy_ssl_options: # ex: no-sslv3 - haproxy_container_image: # ex: haproxy:2.4-dev3-alpine - keepalived_container_image: # ex: arcts/keepalived:1.2.2 - -where the properties of this service specification are: - -* ``service_type`` - Mandatory and set to "ha-rgw" -* ``service_id`` - The name of the service. -* ``placement hosts`` - The hosts where it is desired to run the HA daemons. An HAProxy and a - Keepalived containers will be deployed in these hosts. - The RGW daemons can run in other different hosts or not. -* ``virtual_ip_interface`` - The physical network interface where the virtual ip will be binded -* ``virtual_ip_address`` - The virtual IP ( and network ) where the HA RGW service will be available. - All your RGW clients must point to this IP in order to use the HA RGW - service . -* ``frontend_port`` - The port used to access the HA RGW service -* ``ha_proxy_port`` - The port used by HAProxy containers -* ``ha_proxy_stats_enabled`` - If it is desired to enable the statistics URL in HAProxy daemons -* ``ha_proxy_stats_user`` - User needed to access the HAProxy statistics URL -* ``ha_proxy_stats_password`` - The password needed to access the HAProxy statistics URL -* ``ha_proxy_enable_prometheus_exporter`` - If it is desired to enable the Promethes exporter in HAProxy. This will - allow to consume RGW Service metrics from Grafana. -* ``ha_proxy_monitor_uri``: - To set the API endpoint where the health of HAProxy daemon is provided -* ``keepalived_user`` - User needed to access keepalived daemons -* ``keepalived_password``: - The password needed to access keepalived daemons -* ``ha_proxy_frontend_ssl_certificate``: - SSl certificate. You must paste the content of your .pem file -* ``ha_proxy_frontend_ssl_port``: - The https port used by HAProxy containers -* ``ha_proxy_ssl_dh_param``: - Value used for the `tune.ssl.default-dh-param` setting in the HAProxy - config file -* ``ha_proxy_ssl_ciphers``: - Value used for the `ssl-default-bind-ciphers` setting in HAProxy config - file. -* ``ha_proxy_ssl_options``: - Value used for the `ssl-default-bind-options` setting in HAProxy config - file. -* ``haproxy_container_image``: - HAProxy image location used to pull the image -* ``keepalived_container_image``: - Keepalived image location used to pull the image - -**Useful hints for the RGW Service:** - -* Good to have at least 3 RGW daemons -* Use at least 3 hosts for the HAProxy for RGW service -* In each host an HAProxy and a Keepalived daemon will be deployed. These - daemons can be managed as systemd services - - Deploying custom containers =========================== -- 2.39.5