--- /dev/null
+===============================================================================
+A Detailed Documentation on How to Set up Ceph Kerberos Authentication
+===============================================================================
+
+This document provides details on the Kerberos authorization protocol. This is
+the 1st draft and we will try to keep it updated along with code changes that
+might take place.
+
+Several free implementations of this protocol are available (MIT, Heimdal,
+MS...), covering a wide range of operating systems. The Massachusetts
+Institute of Technology (MIT), where Kerberos was originally developed,
+continues to develop their Kerberos package and it is the implementation we
+chose to work with. `MIT Kerberos <http://web.mit.edu/Kerberos/>`_.
+
+Please, provide feedback to Daniel Oliveira (doliveira@suse.com)
+
+*Last update: Dec 3, 2018*
+
+|
+
+Background
+----------
+
+Before we get into *Kerberos details*, let us define a few terms so we can
+understand what to expect from it, *what it can and can't do*:
+
+Directory Services
+ A directory service is a customizable information store that functions as
+ a single point from which users can locate resources and services
+ distributed throughout the network. This customizable information store
+ also gives administrators a single point for managing its objects and their
+ attributes. Although this information store appears as a single point to
+ the users of the network, it is actually most often stored in a distributed
+ form. A directory service consists of at least one *Directory Server and a
+ Directory Client* and are implemented based on *X.500 standards*.
+
+ *OpenLDAP, 389 Directory Server, MS Active Directory, NetIQ eDirectory* are
+ some good examples.
+
+ A directory service is often characterized as a *write-once-read-many-times
+ service*, meaning the data that would normally be stored in an directory
+ service would not be expected to change on every access.
+
+ The database that forms a directory service *is not designed for
+ transactional data*.
+
+|
+
+LDAP (Lightweight Directory Access Protocol v3)
+ LDAP is a set of LDAP Protocol Exchanges *(not an implementation of a
+ server)* that defines the method by which data is accessed. LDAPv3 is a
+ standard defined by the IETF in RFC 2251 and describes how data is
+ represented in the Directory Service (the Data Model or DIT).
+
+ Finally, it defines how data is loaded into (imported) and saved from
+ (exported) a directory service (using LDIF). LDAP does not define how data
+ is stored or manipulated. Data Store is an 'automagic' process as far as
+ the standard is concerned and is generally handled by back-end modules.
+
+ No Directory Service implementation has all the features of LDAP v3
+ protocol implemented. All Directory Server implementations have their
+ different problems and/or anomalies, and features that may not return
+ results as another Directory Server implementation would.
+
+|
+
+Authentication
+ Authentication is about validating credentials (like User Name/ID and
+ password) to verify the identity. The system determines whether one is what
+ they say they are using their credentials.
+
+ Usually, authentication is done by a username and password, and sometimes
+ in conjunction with *(single, two, or multi) factors of authentication*,
+ which refers to the various ways to be authenticated.
+
+|
+
+Authorization
+ Authorization occurs after the identity is successfully authenticated by
+ the system, which ultimately gives one full permission to access the
+ resources such as information, files, databases, and so forth, almost
+ anything. It determines the ability to access the system and up to what
+ extent (what kind of permissions/rights are given and to where/what).
+
+|
+
+Auditing
+ Auditing takes the results from both *authentication and authorization* and
+ records them into an audit log. The audit log records records all actions
+ taking by/during the authentication and authorization for later review by
+ the administrators. While authentication and authorization are preventive
+ systems (in which unauthorized access is prevented), auditing is a reactive
+ system (in which it gives detailed log of how/when/where someone accessed
+ the environment).
+
+|
+
+Kerberos (KRB v5)
+ Kerberos is a network *authentication protocol*. It is designed to provide
+ strong authentication for client/server applications by using secret-key
+ cryptography (symmetric key). A free implementation of this protocol is
+ available from the MIT. However, Kerberos is available in many commercial
+ products as well.
+
+ It was designed to provide secure authentication to services over an
+ insecure network. Kerberos uses tickets to authenticate a user, or service
+ application and never transmits passwords over the network in the clear.
+ So both client and server can prove their identity without sending any
+ unencrypted secrets over the network.
+
+ Kerberos can be used for single sign-on (SSO). The idea behind SSO is
+ simple, we want to login just once and be able to use any service that we
+ are entitled to, without having to login on each of those services.
+
+|
+
+Simple Authentication and Security Layer (SASL)
+ SASL **(RFC 4422)** is a framework that helps developers to implement
+ different authentication mechanisms (implementing a series of challenges
+ and responses), allowing both clients and servers to negotiate a mutually
+ acceptable mechanism for each connection, instead of hard-coding them.
+
+ Examples of SASL mechanisms:
+
+ * ANONYMOUS **(RFC 4505)**
+
+ - For guest access, meaning *unauthenticated*
+
+ * CRAM-MD5 **(RFC 2195)**
+
+ - Simple challenge-response scheme based on *HMAC-MD5*.
+ It does not establish any security layer. *Less secure than
+ DIGEST-MD5 and GSSAPI.*
+
+ * DIGEST-MD5 **(RFC 2831)**
+
+ - HTTP Digest compatible *(partially)* challenge-response scheme
+ based upon MD5, offering a *data security layer*. It is preferred
+ over PLAIN text passwords, protecting against plain text attacks.
+ It is a mandatory authentication method for LDAPv3 servers.
+
+ * EXTERNAL **(RFCs 4422, 5246, 4301, 2119)**
+
+ - Where *authentication is implicit* in the context (i.e; for
+ protocols using IPsec or TLS [TLS/SSL to performing certificate-
+ based authentication] already). This method uses public keys for
+ strong authentication.
+
+ * GS2 **(RFC 5801)**
+
+ - Family of mechanisms supports arbitrary GSS-API mechanisms in
+ SASL
+
+ * NTLM (MS Proprietary)
+
+ - MS Windows NT LAN Manager authentication mechanism
+
+ * OAuth 1.0/2.0 **(RFCs 5849, 6749, 7628)**
+
+ - Authentication protocol for delegated resource access
+
+ * OTP **(RFC 2444)**
+
+ - One-time password mechanism *(obsoletes the SKEY mechanism)*
+
+ * PLAIN **(RFC 4616)**
+
+ - Simple Cleartext password mechanism **(RFC 4616)**. This is not a
+ preferred mechanism for most applications because of its relative
+ lack of strength.
+
+ * SCRAM **(RFCs 5802, 7677)**
+
+ - Modern challenge-response scheme based mechanism with channel
+ binding support
+
+|
+
+Generic Security Services Application Program Interface (GSSAPI)
+ GSSAPI **(RFCs 2078, 2743, 2744, 4121, 4752)** is widely used by protocol
+ implementers as a way to implement Kerberos v5 support in their
+ applications. It provides a generic interface and message format that can
+ encapsulate authentication exchanges from any authentication method that
+ has a GSSAPI-compliant library.
+
+ It does not define a protocol, authentication, or security mechanism
+ itself; it instead makes it easier for application programmers to support
+ multiple authentication mechanisms by providing a uniform, generic API for
+ security services. It is a set of functions that include both an API and a
+ methodology for approaching authentication, aiming to insulate application
+ protocols from the specifics of security protocols as much as possible.
+
+ *Microsoft Windows Kerberos* implementation does not include GSSAPI support
+ but instead includes a *Microsoft-specific API*, the *Security Support
+ Provider Interface (SSPI)*. In Windows, an SSPI client can communicate with
+ a *GSSAPI server*.
+
+ *Most applications that support GSSAPI also support Kerberos v5.*
+
+|
+
+Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO)
+ As we can see, GSSAPI solves the problem of providing a single API to
+ different authentication mechanisms. However, it does not solve the problem
+ of negotiating which mechanism to use. In fact for GSSAPI to work, the two
+ applications communicating with each other must know in advance what
+ authentication mechanism they plan to use, which usually is not a problem
+ if only one mechanism is supported (meaning Kerberos v5).
+
+ However, if there are multiple mechanisms to choose from, a method is
+ needed to securely negotiate an authentication mechanism that is mutually
+ supported between both client and server; which is where
+ *SPNEGO (RFC 2478, 4178)* makes a difference.
+
+ *SPNEGO* provides a framework for two parties that are engaged in
+ authentication to select from a set of possible authentication mechanisms,
+ in a manner that preserves the opaque nature of the security protocols to
+ the application protocol that uses it.
+
+ It is a security protocol that uses a *GSSAPI authentication mechanism* and
+ negotiates among several available authentication mechanisms in an
+ implementation, selecting one for use to satisfy the authentication needs
+ of the application protocol.
+
+ It is a *meta protocol* that travels entirely in other application
+ protocols; it is never used directly without an application protocol.
+
+|
+
+*Why is this important and why do we care? Like, at all?*
+
+ Having this background information in mind, we can easily describe things
+ like:
+
+ 1. *Ceph Kerberos authentication* is based totally on MIT *Kerberos*
+ implementation using *GSSAPI*.
+
+ 2. At the moment we are still using *Kerberos default backend
+ database*, however we plan on adding LDAP as a backend which would
+ provide us with *authentication with GSSAPI (KRB5)* and *authorization
+ with LDAP (LDAPv3)*, via *SASL mechanism*.
+
+|
+
+Before We Start
+---------------
+
+We assume the environment already has some external services up and running
+properly:
+
+ * Kerberos needs to be properly configured, which also means (for both
+ every server and KDC):
+
+ - Time Synchronization (either using `NTP <http://www.ntp.org/>`_ or `chrony <https://chrony.tuxfamily.org/>`_).
+
+ + Not only Kerberos, but also Ceph depends and relies on time
+ synchronization.
+
+ - DNS resolution
+
+ + Both *(forward and reverse)* zones, with *fully qualified domain
+ name (fqdn)* ``(hostname + domain.name)``
+
+ + KDC discover can be set up to to use DNS ``(srv resources)`` as
+ service location protocol *(RFCs 2052, 2782)*, as well as *host
+ or domain* to the *appropriate realm* ``(txt record)``.
+
+ + Even though these DNS entries/settings are not required to run a
+ ``Kerberos realm``, they certainly help to eliminate the need for
+ manual configuration on all clients.
+
+ + This is extremely important, once most of the Kerberos issues are
+ usually related to name resolution. Kerberos is very picky when
+ checking on systems names and host lookups.
+
+ * Whenever possible, in order to avoid a *single point of failure*, set up
+ a *backup, secondary, or slave*, for every piece/part in the
+ infrastructure ``(ntp, dns, and kdc servers)``.
+
+
+Also, the following *Kerberos terminology* is important:
+
+ * Ticket
+
+ - Tickets or Credentials, are a set of information that can be used to
+ verify the client's identity. Kerberos tickets may be stored in a
+ file, or they may exist only in memory.
+
+ - The first ticket obtained is a ticket-granting ticket (TGT), which
+ allows the clients to obtain additional tickets. These additional
+ tickets give the client permission for specific services. The
+ requesting and granting of these additional tickets happens
+ transparently.
+
+ + The TGT, which expires at a specified time, permits the client to
+ obtain additional tickets, which give permission for specific
+ services. The requesting and granting of these additional tickets
+ is user-transparent.
+
+ * Key Distribution Center (KDC).
+
+ - The KDC creates a ticket-granting ticket (TGT) for the client,
+ encrypts it using the client's password as the key, and sends the
+ encrypted TGT back to the client. The client then attempts to decrypt
+ the TGT, using its password. If the client successfully decrypts the
+ TGT (i.e., if the client gave the correct password), it keeps the
+ decrypted TGT, which indicates proof of the client's identity.
+
+ - The KDC is comprised of three components:
+
+ + Kerberos database, which stores all the information about the
+ principals and the realm they belong to, among other things.
+ + Authentication service (AS)
+ + Ticket-granting service (TGS)
+
+ * Client
+
+ - Either a *user, host or a service* who sends a request for a ticket.
+
+ * Principal
+
+ - It is a unique identity to which Kerberos can assign tickets.
+ Principals can have an arbitrary number of components. Each component
+ is separated by a component separator, generally ``/``. The last
+ component is the *realm*, separated from the rest of the principal by
+ the realm separator, generally ``@``.
+
+ - If there is no realm component in the principal, then it will be
+ assumed that the principal is in the default realm for the context in
+ which it is being used.
+
+ - Usually, a principal is divided into three parts:
+
+ + The ``primary``, the ``instance``, and the ``realm``
+
+ + The format of a typical Kerberos V5 principal is
+ ``primary/instance@REALM``.
+
+ + The ``primary`` is the first part of the principal. In the case
+ of a user, it's the same as the ``username``. For a host, the
+ primary is the word ``host``. For Ceph, will use ``ceph`` as a
+ primary name which makes it easier to organize and identify Ceph
+ related principals.
+
+ + The ``instance`` is an optional string that qualifies the
+ primary. The instance is separated from the primary by a slash
+ ``/``. In the case of a user, the instance is usually ``null``,
+ but a user might also have an additional principal, with an
+ instance called ``admin``, which one uses to administrate a
+ database.
+
+ The principal ``johndoe@MYDOMAIN.COM`` is completely separate
+ from the principal ``johndoe/admin@MYDOMAIN.COM``, with a
+ separate password, and separate permissions. In the case of a
+ host, the instance is the fully qualified hostname,
+ i.e., ``osd1.MYDOMAIN.COM``.
+
+ + The ``realm`` is the Kerberos realm. Usually, the Kerberos realm
+ is the domain name, in *upper-case letters*. For example, the
+ machine ``osd1.MYDOMAIN.COM`` would be in the realm
+ ``MYDOMAIN.COM``.
+
+ * Keytab
+
+ - A keytab file stores the actual encryption key that can be used in
+ lieu of a password challenge for a given principal. Creating keytab
+ files are useful for noninteractive principals, such as *Service
+ Principal Names*, which are often associated with long-running
+ processes like Ceph daemons. A keytab file does not have to be a
+ "1:1 mapping" to a single principal. Multiple different principal
+ keys can be stored in a single keytab file:
+
+ + The keytab file allows a user/service to authenticate without
+ knowledge of the password. Due to this, *keytabs should be
+ protected* with appropriate controls to prevent unauthorized
+ users from authenticating with it.
+
+ + The default client keytab file is ``/etc/krb5.keytab``
+
+|
+
+The 'Ceph side' of the things
+------------------------------
+
+In order to configure connections (from Ceph nodes) to the KDC:
+
+1. Login to the Kerberos client (Ceph server nodes) and confirm it is properly
+ configured, by checking and editing ``/etc/krb5.conf`` file properly: ::
+
+ /etc/krb5.conf
+ [libdefaults]
+ dns_canonicalize_hostname = false
+ rdns = false
+ forwardable = true
+ dns_lookup_realm = true
+ dns_lookup_kdc = true
+ allow_weak_crypto = false
+ default_realm = MYDOMAIN.COM
+ default_ccache_name = KEYRING:persistent:%{uid}
+ [realms]
+ MYDOMAIN.COM = {
+ kdc = kerberos.mydomain.com
+ admin_server = kerberos.mydomain.com
+ ...
+ }
+ ...
+
+
+2. Login to the *KDC Server* and confirm it is properly configured to
+ authenticate to the Kerberos realm in question:
+
+ a. Kerberos related DNS RRs: ::
+
+ /var/lib/named/master/mydomain.com
+ kerberos IN A 192.168.10.21
+ kerberos-slave IN A 192.168.10.22
+ _kerberos IN TXT "MYDOMAIN.COM"
+ _kerberos._udp IN SRV 1 0 88 kerberos
+ _kerberos._tcp IN SRV 1 0 88 kerberos
+ _kerberos._udp IN SRV 20 0 88 kerberos-slave
+ _kerberos-master._udp IN SRV 0 0 88 kerberos
+ _kerberos-adm._tcp IN SRV 0 0 749 kerberos
+ _kpasswd._udp IN SRV 0 0 464 kerberos
+ ...
+
+
+ b. KDC configuration file: ::
+
+ /var/lib/kerberos/krb5kdc/kdc.conf
+ [kdcdefaults]
+ kdc_ports = 750,88
+ [realms]
+ MYDOMAIN.COM = {
+ acl_file = /var/lib/kerberos/krb5kdc/kadm5.acl
+ admin_keytab = FILE:/var/lib/kerberos/krb5kdc/kadm5.keytab
+ default_principal_flags = +postdateable +forwardable +renewable +proxiable
+ +dup-skey -preauth -hwauth +service
+ +tgt-based +allow-tickets -pwchange
+ -pwservice
+ dict_file = /var/lib/kerberos/krb5kdc/kadm5.dict
+ key_stash_file = /var/lib/kerberos/krb5kdc/.k5.MYDOMAIN.COM
+ kdc_ports = 750,88
+ max_life = 0d 10h 0m 0s
+ max_renewable_life = 7d 0h 0m 0s
+ }
+ ...
+
+
+3. Still on the KDC Server, run the Kerberos administration utility;
+ ``kadmin.local`` so we can list all the principals already created. ::
+
+ kadmin.local: listprincs
+ K/M@MYDOMAIN.COM
+ krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
+ kadmin/admin@MYDOMAIN.COM
+ kadmin/changepw@MYDOMAIN.COM
+ kadmin/history@MYDOMAIN.COM
+ kadmin/kerberos.mydomain.com@MYDOMAIN.COM
+ root/admin@MYDOMAIN.COM
+ ...
+
+
+4. Add a *principal for each Ceph cluster node* we want to be authenticated by
+ Kerberos:
+
+ a. Adding principals: ::
+
+ kadmin.local: addprinc -randkey ceph/ceph-mon1
+ Principal "ceph/ceph-mon1@MYDOMAIN.COM" created.
+ kadmin.local: addprinc -randkey ceph/ceph-osd1
+ Principal "ceph/ceph-osd1@MYDOMAIN.COM" created.
+ kadmin.local: addprinc -randkey ceph/ceph-osd2
+ Principal "ceph/ceph-osd2@MYDOMAIN.COM" created.
+ kadmin.local: addprinc -randkey ceph/ceph-osd3
+ Principal "ceph/ceph-osd3@MYDOMAIN.COM" created.
+ kadmin.local: addprinc -randkey ceph/ceph-osd4
+ Principal "ceph/ceph-osd4@MYDOMAIN.COM" created.
+ kadmin.local: listprincs
+ K/M@MYDOMAIN.COM
+ krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
+ kadmin/admin@MYDOMAIN.COM
+ kadmin/changepw@MYDOMAIN.COM
+ kadmin/history@MYDOMAIN.COM
+ kadmin/kerberos.mydomain.com@MYDOMAIN.COM
+ root/admin@MYDOMAIN.COM
+ ceph/ceph-mon1@MYDOMAIN.COM
+ ceph/ceph-osd1@MYDOMAIN.COM
+ ceph/ceph-osd2@MYDOMAIN.COM
+ ceph/ceph-osd3@MYDOMAIN.COM
+ ceph/ceph-osd4@MYDOMAIN.COM
+ ...
+
+
+ b. This follows the same idea if we are creating a *user principal* ::
+
+ kadmin.local: addprinc johndoe
+ WARNING: no policy specified for johndoe@MYDOMAIN.COM; defaulting to no policy
+ Enter password for principal "johndoe@MYDOMAIN.COM":
+ Re-enter password for principal "johndoe@MYDOMAIN.COM":
+ Principal "johndoe@MYDOMAIN.COM" created.
+ ...
+
+
+5. Create a *keytab file* for each Ceph cluster node:
+
+ As the default client keytab file is ``/etc/krb5.keytab``, we will want to
+ use a different file name, so we especify which *keytab file to create* and
+ which *principal to export keys* from: ::
+
+ kadmin.local: ktadd -k /etc/gss_client_mon1.ktab ceph/ceph-mon1
+ Entry for principal ceph/ceph-mon1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_mon1.ktab.
+ Entry for principal ceph/ceph-mon1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_mon1.ktab.
+ Entry for principal ceph/ceph-mon1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/gss_client_mon1.ktab.
+ Entry for principal ceph/ceph-mon1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/etc/gss_client_mon1.ktab.
+ kadmin.local: ktadd -k /etc/gss_client_osd1.ktab ceph/ceph-osd1
+ Entry for principal ceph/ceph-osd1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd1.ktab.
+ Entry for principal ceph/ceph-osd1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd1.ktab.
+ Entry for principal ceph/ceph-osd1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/gss_client_osd1.ktab.
+ Entry for principal ceph/ceph-osd1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/etc/gss_client_osd1.ktab.
+ kadmin.local: ktadd -k /etc/gss_client_osd2.ktab ceph/ceph-osd2
+ Entry for principal ceph/ceph-osd2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd2.ktab.
+ Entry for principal ceph/ceph-osd2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd2.ktab.
+ Entry for principal ceph/ceph-osd2 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/gss_client_osd2.ktab.
+ Entry for principal ceph/ceph-osd2 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/etc/gss_client_osd2.ktab.
+ kadmin.local: ktadd -k /etc/gss_client_osd3.ktab ceph/ceph-osd3
+ Entry for principal ceph/ceph-osd3 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd3.ktab.
+ Entry for principal ceph/ceph-osd3 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd3.ktab.
+ Entry for principal ceph/ceph-osd3 with kvno 3, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/gss_client_osd3.ktab.
+ Entry for principal ceph/ceph-osd3 with kvno 3, encryption type arcfour-hmac added to keytab WRFILE:/etc/gss_client_osd3.ktab.
+ kadmin.local: ktadd -k /etc/gss_client_osd4.ktab ceph/ceph-osd4
+ Entry for principal ceph/ceph-osd4 with kvno 4, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd4.ktab.
+ Entry for principal ceph/ceph-osd4 with kvno 4, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd4.ktab.
+ Entry for principal ceph/ceph-osd4 with kvno 4, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/gss_client_osd4.ktab.
+ Entry for principal ceph/ceph-osd4 with kvno 4, encryption type arcfour-hmac added to keytab WRFILE:/etc/gss_client_osd4.ktab.
+
+ # ls -1 /etc/gss_client_*
+ /etc/gss_client_mon1.ktab
+ /etc/gss_client_osd1.ktab
+ /etc/gss_client_osd2.ktab
+ /etc/gss_client_osd3.ktab
+ /etc/gss_client_osd4.ktab
+
+
+ We can also check these newly created keytab client files by: ::
+
+ # klist -kte /etc/gss_client_mon1.ktab
+ Keytab name: FILE:/etc/gss_client_mon1.ktab
+ KVNO Timestamp Principal
+ ---- ------------------- ------------------------------------------------------
+ 2 10/8/2018 14:35:30 ceph/ceph-mon1@MYDOMAIN.COM (aes256-cts-hmac-sha1-96)
+ 2 10/8/2018 14:35:31 ceph/ceph-mon1@MYDOMAIN.COM (aes128-cts-hmac-sha1-96)
+ 2 10/8/2018 14:35:31 ceph/ceph-mon1@MYDOMAIN.COM (des3-cbc-sha1)
+ 2 10/8/2018 14:35:31 ceph/ceph-mon1@MYDOMAIN.COM (arcfour-hmac)
+ ...
+
+
+6. A new *set parameter* was added in Ceph, ``gss ktab client file`` which
+ points to the keytab file related to the Ceph node *(or principal)* in
+ question.
+
+ By default it points to ``/var/lib/ceph/$name/gss_client_$name.ktab``. So,
+ in the case of a Ceph server ``osd1.mydomain.com``, the location and name
+ of the keytab file should be: ``/var/lib/ceph/osd1/gss_client_osd1.ktab``
+
+ Therefore, we need to ``scp`` each of these newly created keytab files from
+ the KDC to their respective Ceph cluster nodes (i.e):
+ ``# for node in mon1 osd1 osd2 osd3 osd4; do scp /etc/gss_client_$node*.ktab root@ceph-$node:/var/lib/ceph/$node/; done``
+
+ Or whatever other way one feels comfortable with, as long as each keytab
+ client file gets copied over to the proper location.
+
+ At this point, even *without using any keytab client file* we should be
+ already able to authenticate a *user principal*: ::
+
+ # kdestroy -A && kinit -f johndoe && klist -f
+ Password for johndoe@MYDOMAIN.COM:
+ Ticket cache: KEYRING:persistent:0:0
+ Default principal: johndoe@MYDOMAIN.COM
+
+ Valid starting Expires Service principal
+ 10/10/2018 15:32:01 10/11/2018 07:32:01 krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
+ renew until 10/11/2018 15:32:01, Flags: FRI
+ ...
+
+
+ Given that the *keytab client file* is/should already be copied and available at the
+ Kerberos client (Ceph cluster node), we should be able to athenticate using it before
+ going forward: ::
+
+ # kdestroy -A && kinit -k -t /etc/gss_client_mon1.ktab -f 'ceph/ceph-mon1@MYDOMAIN.COM' && klist -f
+ Ticket cache: KEYRING:persistent:0:0
+ Default principal: ceph/ceph-mon1@MYDOMAIN.COM
+
+ Valid starting Expires Service principal
+ 10/10/2018 15:54:25 10/11/2018 07:54:25 krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
+ renew until 10/11/2018 15:54:25, Flags: FRI
+ ...
+
+
+7. The default client keytab is used, if it is present and readable, to
+ automatically obtain initial credentials for GSSAPI client applications. The
+ principal name of the first entry in the client keytab is used by default
+ when obtaining initial credentials:
+
+ a. The ``KRB5_CLIENT_KTNAME environment`` variable.
+ b. The ``default_client_keytab_name`` profile variable in ``[libdefaults]``.
+ c. The hardcoded default, ``DEFCKTNAME``.
+
+ So, what we do is to internally, set the environment variable
+ ``KRB5_CLIENT_KTNAME`` to the same location as ``gss_ktab_client_file``,
+ so ``/var/lib/ceph/osd1/gss_client_osd1.ktab``, and change the ``ceph.conf``
+ file to add the new authentication method. ::
+
+ /etc/ceph/ceph.conf
+ [global]
+ ...
+ auth cluster required = gss
+ auth service required = gss
+ auth client required = gss
+ gss ktab client file = /{$my_new_location}/{$my_new_ktab_client_file.keytab}
+ ...
+
+
+8. With that the GSSAPIs will then be able to read the keytab file and using
+ the process of name and service resolution *(provided by the DNS)*, able to
+ request a *TGT* as follows:
+
+ a. User/Client sends principal identity and credentials to the KDC Server
+ (TGT request).
+ b. KDC checks its internal database for the principal in question.
+ c. a TGT is created and wrapped by the KDC, using the principal's key
+ (TGT + Key).
+ d. The newly created TGT, is decrypted and stored in the credentials
+ cache.
+ e. At this point, Kerberos/GSSAPI aware applications (and/or services) are
+ able to check the list of active TGT in the keytab file.
+
+|
+|
+
+** *For Ceph Developers Only* **
+=================================
+
+We certainly could have used straight native ``KRB5 APIs`` (instead of
+``GSSAPIs``), but we wanted a more portable option as regards network security,
+which is the hallmark of the ``GSS`` *(Generic Security Standard)* ``-API``.
+It does not actually provide security services itself.
+
+Rather, it is a framework that provides security services to callers in a
+generic way. ::
+
+ +---------------------------------+
+ | Application |
+ +---------------------------------+
+ | Protocol (RPC, Etc. [Optional]) |
+ +---------------------------------+
+ | GSS-API |
+ +---------------------------------+
+ | Security Mechs (Krb v5, Etc) |
+ +---------------------------------+
+
+
+The GSS-API does two main things:
+
+ 1. It creates a security context in which data can be passed between
+ applications. A context can be thought of as a sort of *"state of trust"*
+ between two applications.
+
+ Applications that share a context know who each other are and thus can
+ permit data transfers between them as long as the context lasts.
+
+ 2. It applies one or more types of protection, known as *"security services"*,
+ to the data to be transmitted.
+
+
+GSS-API provides several types of portability for applications:
+
+ a. **Mechanism independence.** GSS-API provides a generic interface to the
+ mechanisms for which it has been implemented. By specifying a default
+ security mechanism, an application does not need to know which mechanism
+ it is using (for example, Kerberos v5), or even what type of mechanism
+ it uses. As an example, when an application forwards a user's credential
+ to a server, it does not need to know if that credential has a Kerberos
+ format or the format used by some other mechanism, nor how the
+ credentials are stored by the mechanism and accessed by the application.
+ (If necessary, an application can specify a particular mechanism to use)
+
+ b. **Protocol independence.** The GSS-API is independent of any
+ communications protocol or protocol suite. It can be used with
+ applications that use, for example, sockets, RCP, or TCP/IP.
+ RPCSEC_GSS "RPCSEC_GSS Layer" is an additional layer that smoothly
+ integrates GSS-API with RPC.
+
+ c. **Platform independence.** The GSS-API is completely oblivious to the
+ type of operating system on which an application is running.
+
+ d. **Quality of Protection independence.** Quality of Protection (QOP) is
+ the name given to the type of algorithm used in encrypting data or
+ generating cryptographic tags; the GSS-API allows a programmer to ignore
+ QOP, using a default provided by the GSS-API.
+ (On the other hand, an application can specify the QOP if necessary.)
+
+ The basic security offered by the GSS-API is authentication. Authentication
+ is the verification of an identity: *if you are authenticated, it means
+ that you are recognized to be who you say you are.*
+
+ The GSS-API provides for two additional security services, if supported by the
+ underlying mechanisms:
+
+ 1. **Integrity:** It's not always sufficient to know that an application
+ sending you data is who it claims to be. The data itself could have
+ become corrupted or compromised.
+
+ The GSS-API provides for data to be accompanied by a cryptographic tag,
+ known as an ``Message Integrity Code (MIC)``, to prove that the data
+ that arrives at your doorstep is the same as the data that the sender
+ transmitted. This verification of the data's validity is known as
+ *"integrity"*.
+
+ 2. **Confidentiality:** Both authentication and integrity, however, leave
+ the data itself alone, so if it's somehow intercepted, others can read
+ it.
+
+ The GSS-API therefore allows data to be encrypted, if underlying
+ mechanisms support it. This encryption of data is known as *"confidentiality"*.
+
+|
+
+Mechanisms Available With GSS-API:
+
+ The current implementation of the GSS-API works only with the Kerberos v5 security
+ mechanism. ::
+
+ Mechanism Name Object Identifier Shared Library Kernel Module
+ ---------------------- ---------------------- -------------- --------------
+ diffie_hellman_640_0 1.3.6.4.1.42.2.26.2.4 dh640-0.so.1
+ diffie_hellman_1024_0 1.3.6.4.1.42.2.26.2.5 dh1024-0.so.1
+ SPNEGO 1.3.6.1.5.5.2
+ iakerb 1.3.6.1.5.2.5
+ SCRAM-SHA-1 1.3.6.1.5.5.14
+ SCRAM-SHA-256 1.3.6.1.5.5.18
+ GSS-EAP (arc) 1.3.6.1.5.5.15.1.1.*
+ kerberos_v5 1.2.840.113554.1.2.2 gl/mech_krb5.so gl_kmech_krb5
+
+ Therefore:
+ Kerberos Version 5 GSS-API Mechanism
+ OID {1.2.840.113554.1.2.2}
+
+ Kerberos Version 5 GSS-API Mechanism
+ Simple and Protected GSS-API Negotiation Mechanism
+ OID {1.3.6.1.5.5.2}
+
+
+ There are two different formats:
+
+ 1. The first, ``{ 1 2 3 4 }``, is officially mandated by the GSS-API
+ specs. ``gss_str_to_oid()`` expects this first format.
+
+ 2. The second, ``1.2.3.4``, is more widely used but is not an official
+ standard format.
+
+ Although the GSS-API makes protecting data simple, it does not do certain
+ things, in order to maximize its generic nature. These include:
+
+ a. Provide security credentials for a user or application. These must
+ be provided by the underlying security mechanism(s). The GSS-API
+ does allow applications to acquire credentials, either automatically
+ or explicitly.
+
+ b. Transfer data between applications. It is the application's
+ responsibility to handle the transfer of all data between peers,
+ whether it is security-related or "plain" data.
+
+ c. Distinguish between different types of transmitted data (for
+ example, to know or determine that a data packet is plain data and
+ not GSS-API related).
+
+ d. Indicate status due to remote (asynchronous) errors.
+
+ e. Automatically protect information sent between processes of a
+ multiprocess program.
+
+ f. Allocate string buffers ("Strings and Similar Data") to be passed to
+ GSS-API functions.
+
+ g. Deallocate GSS-API data spaces. These must be explicitly deallocated
+ with functions such as ``gss_release_buffer()`` and
+ ``gss_delete_name()``.
+
+|
+
+These are the basic steps in using the GSS-API:
+
+ 1. Each application, sender and recipient, acquires credentials explicitly,
+ if credentials have not been acquired automatically.
+
+ 2. The sender initiates a security context and the recipient accepts it.
+
+ 3. The sender applies security protection to the message (data) it wants to
+ transmit. This means that it either encrypts the message or stamps it
+ with an identification tag. The sender transmits the protected message.
+ (The sender can choose not to apply either security protection, in which
+ case the message has only the default GSS-API security service
+ associated with it. That is authentication, in which the recipient knows
+ that the sender is who it claims to be.)
+
+ 4. The recipient decrypts the message (if needed) and verifies it
+ (if appropriate).
+
+ 5. (Optional) The recipient returns an identification tag to the sender for
+ confirmation.
+
+ 6. Both applications destroy the shared security context. If necessary,
+ they can also deallocate any *"leftover"* GSS-API data.
+
+ Applications that use the GSS-API should include the file ``gssapi.h``.
+
+ Good References:
+ - `rfc1964 <https://tools.ietf.org/html/rfc1964>`_.
+ - `rfc2743 <https://tools.ietf.org/html/rfc2743>`_.
+ - `rfc2744 <https://tools.ietf.org/html/rfc2744>`_.
+ - `rfc4178 <https://tools.ietf.org/html/rfc4178>`_.
+ - `rfc6649 <https://tools.ietf.org/html/rfc6649>`_.
+ - `MIT Kerberos Documentation <https://web.mit.edu/kerberos/krb5-latest/doc/appdev/gssapi.html>`_.
+
+|
+
+** *Kerberos Server Setup* **
+------------------------------
+
+First and foremost, ``this is not a recommendation for a production
+environment``. We are not covering ``Master/Slave replication cluster`` or
+anything production environment related (*ntp/chrony, dns, pam/nss, sssd, etc*).
+
+Also, on the server side there might be different dependencies and/or
+configuration steps needed, depending on which backend database will be used.
+``LDAP as a backend database`` is a good example of that.
+
+On the client side there are different steps depending on which client backend
+configuration will be used. For example ``PAM/NSS`` or ``SSSD`` (along with
+LDAP for identity service, [and Kerberos for authentication service]) which is
+the best suited option for joining ``MS Active Directory domains``, and doing
+``User Logon Management``.
+
+By no means we intend to cover every possible scenario/combination here. These
+steps are for a simple *get a (MIT) Kerberos Server up and running*.
+
+Please, note that *rpm packages might have slightly different names*, as well
+as the locations for the binaries and/or configuration files, depending on
+which Linux distro we are referring to.
+
+Finally, keep in mind that some Linux distros will have their own ``wizards``,
+which can perform the basic needed configuration: ::
+
+ SUSE:
+ Kerberos server:
+ yast2 auth-server
+
+ Kerberos client:
+ pam/nss: yast2 ldapkrb
+ sssd: yast2 auth-client
+
+
+However, we are going through the ``manual configuration``.
+
+
+In order to get a new MIT KDC Server running:
+
+1. Install the KDC server by:
+
+ a. Install the needed packages: ::
+
+ SUSE: zypper install krb5 krb5-server krb5-client
+ Additionally:
+ for development: krb5-devel
+ if using 'sssd': sssd-krb5 sssd-krb5-common
+
+ REDHAT: yum install krb5-server krb5-libs krb5-workstation
+ Additionally: 'Needs to be checked'
+
+
+ b. Edit the KDC Server configuration file: ::
+
+ /var/lib/kerberos/krb5kdc/kdc.conf
+ [kdcdefaults]
+ kdc_ports = 750,88
+ [realms]
+ MYDOMAIN.COM = {
+ acl_file = /var/lib/kerberos/krb5kdc/kadm5.acl
+ admin_keytab = FILE:/var/lib/kerberos/krb5kdc/kadm5.keytab
+ default_principal_flags = +postdateable +forwardable +renewable +proxiable
+ +dup-skey -preauth -hwauth +service
+ +tgt-based +allow-tickets -pwchange
+ -pwservice
+ dict_file = /var/lib/kerberos/krb5kdc/kadm5.dict
+ key_stash_file = /var/lib/kerberos/krb5kdc/.k5.MYDOMAIN.COM
+ kdc_ports = 750,88
+ max_life = 0d 10h 0m 0s
+ max_renewable_life = 7d 0h 0m 0s
+ }
+ ...
+
+
+ c. Edit the Kerberos Client configuration file: ::
+
+ /etc/krb5.conf
+ [libdefaults]
+ dns_canonicalize_hostname = false
+ rdns = false
+ forwardable = true
+ dns_lookup_realm = true //--> if using DNS/DNSMasq
+ dns_lookup_kdc = true //--> if using DNS/DNSMasq
+ allow_weak_crypto = false
+ default_realm = MYDOMAIN.COM
+ default_ccache_name = KEYRING:persistent:%{uid}
+
+ [realms]
+ MYDOMAIN.COM = {
+ kdc = kerberos.mydomain.com
+ admin_server = kerberos.mydomain.com
+ ...
+ }
+ ...
+
+
+2. Create the Kerberos database: ::
+
+ SUSE: kdb5_util create -s
+
+ REDHAT: kdb5_util create -s
+
+
+3. Enable and Start both 'KDC and KDC admin' servers: ::
+
+ SUSE: systemctl enable/start krb5kdc
+ systemctl enable/start kadmind
+
+ REDHAT: systemctl enable/start krb5kdc
+ systemctl enable/start kadmin
+
+
+4. Create a Kerberos Administrator
+ Kerberos principals can be created either locally on the KDC server itself
+ or through the network, using an 'admin principal'. On the KDC server,
+ using ``kadmin.local``:
+
+ a. List the existing principals: ::
+
+ kadmin.local: listprincs
+ K/M@MYDOMAIN.COM
+ krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
+ kadmin/admin@MYDOMAIN.COM
+ kadmin/changepw@MYDOMAIN.COM
+ kadmin/history@MYDOMAIN.COM
+ kadmin/kerberos.mydomain.com@MYDOMAIN.COM
+ root/admin@MYDOMAIN.COM
+ ...
+
+
+ b. In case we don't have a built-in 'admin principal', we then create one
+ (whatever ``principal name``, we are using ``root``, once by default
+ ``kinit`` tries to authenticate using the same system login user name,
+ unless a ``principal`` is passed as an argument ``kinit principal``): ::
+
+ # kadmin.local -q "addprinc root/admin"
+ Authenticating as principal root/admin@MYDOMAIN.COM with password.
+ WARNING: no policy specified for root/admin@MYDOMAIN.COM; defaulting to no policy
+ Enter password for principal "root/admin@MYDOMAIN.COM":
+
+
+ c. Confirm the newly created 'admin principal' has the needed permissions
+ in the KDC ACL (if ACLs are changed, ``kadmind`` needs to be restarted): ::
+
+ SUSE: /var/lib/kerberos/krb5kdc/kadm5.acl
+ REDHAT: /var/kerberos/krb5kdc/kadm5.acl
+
+ ###############################################################################
+ #Kerberos_principal permissions [target_principal] [restrictions]
+ ###############################################################################
+ #
+ */admin@MYDOMAIN.COM *
+
+
+ d. Create a simple 'user principal' (same steps as by *The 'Ceph side' of
+ the things*; 4a): ::
+
+ kadmin.local: addprinc johndoe
+ WARNING: no policy specified for johndoe@MYDOMAIN.COM; defaulting to no policy
+ Enter password for principal "johndoe@MYDOMAIN.COM":
+ Re-enter password for principal "johndoe@MYDOMAIN.COM":
+ Principal "johndoe@MYDOMAIN.COM" created.
+
+
+ e. Confirm the newly created 'user principal' is able to authenticate (same
+ steps as by *The 'Ceph side' of the things*; 6): ::
+
+ # kdestroy -A && kinit -f johndoe && klist -f
+ Password for johndoe@MYDOMAIN.COM:
+ Ticket cache: KEYRING:persistent:0:0
+ Default principal: johndoe@MYDOMAIN.COM
+
+ Valid starting Expires Service principal
+ 11/16/2018 13:11:16 11/16/2018 23:11:16 krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
+ renew until 11/17/2018 13:11:16, Flags: FRI
+ ...
+
+
+5. At this point, we should have a *simple (MIT) Kerberos Server up and running*:
+
+ a. Considering we will want to work with keytab files, for both 'user and
+ service' principals, refer to The *'Ceph side' of the things* starting
+ at step 4.
+
+ b. Make sure you are comfortable with following and their ``manpages``: ::
+
+ krb5.conf -> Krb client config file
+ kdc.conf -> KDC server config file
+
+ krb5kdc -> KDC server daemon
+ kadmind -> KDC administration daemon
+
+ kadmin -> Krb administration tool
+ kdb5_util -> Krb low-level database administration tool
+
+ kinit -> Obtain and cache Kerberos ticket-granting ticket tool
+ klist -> List cached Kerberos tickets tool
+ kdestroy -> Destroy Kerberos tickets tool
+
+
+6. Name Resolution
+ As mentioned earlier, Kerberos *relies heavly on name resolution*. Most of
+ the Kerberos issues are usually related to name resolution, since Kerberos
+ is *very picky* on both *systems names* and *host lookups*.
+
+ a. As described in *The 'Ceph side' of the things*; step 2a, DNS RRs
+ greatly improves service location and host/domain resolution, by using
+ ``(srv resources)`` and ``(txt record)`` respectively (as per
+ *Before We Start*; *DNS resolution*). ::
+
+ /var/lib/named/master/mydomain.com
+ kerberos IN A 192.168.10.21
+ kerberos-slave IN A 192.168.10.22
+ _kerberos IN TXT "MYDOMAIN.COM"
+ _kerberos._udp IN SRV 1 0 88 kerberos
+ _kerberos._tcp IN SRV 1 0 88 kerberos
+ _kerberos._udp IN SRV 20 0 88 kerberos-slave
+ _kerberos-master._udp IN SRV 0 0 88 kerberos
+ _kerberos-adm._tcp IN SRV 0 0 749 kerberos
+ _kpasswd._udp IN SRV 0 0 464 kerberos
+ ...
+
+
+ b. For a small network or development environment, where a *DNS server is
+ not available*, we have the option to use ``DNSMasq``, an
+ ease-to-configure lightweight DNS server (along with some other
+ capabilities).
+
+ These records can be added to ``/etc/dnsmasq.conf`` (in addition to the
+ needed 'host records'): ::
+
+ /etc/dnsmasq.conf
+ ...
+ txt-record=_kerberos.mydomain.com,"MYDOMAIN.COM"
+ srv-host=_kerberos._udp.mydomain.com,kerberos.mydomain.com,88,1
+ srv-host=_kerberos._udp.mydomain.com,kerberos-2.mydomain.com,88,20
+ srv-host=_kerberos-master._udp.mydomain.com,kerberos.mydomain.com,88,0
+ srv-host=_kerberos-adm._tcp.mydomain.com,kerberos.mydomain.com,749,0
+ srv-host=_kpasswd._udp.mydomain.com,kerberos.mydomain.com,464,0
+ srv-host=_kerberos._tcp.mydomain.com,kerberos.mydomain.com,88,1
+ ...
+
+
+ c. After 'b)' is all set, and ``dnsmasq`` service up and running, we can
+ test it using: ::
+
+ # nslookup kerberos
+ Server: 192.168.10.1
+ Address: 192.168.10.1#53
+
+ Name: kerberos.mydomain.com
+ Address: 192.168.10.21
+
+ # host -t SRV _kerberos._tcp.mydomain.com
+ _kerberos._tcp.mydomain.com has SRV record 1 0 88 kerberos.mydomain.com.
+
+ # host -t SRV {each srv-host record}
+ # host -t TXT _kerberos.mydomain.com
+ _kerberos.mydomain.com descriptive text "MYDOMAIN.COM"
+ ...
+
+
+ f. As long as ``name resolution`` is working properly, either ``dnsmasq``
+ or ``named``, Kerberos should be able to find the needed service
+ records.
projects / ceph-ci.git / commitdiff