12/20/2019

Configuring Authentication with Kerberos in Kafka

One day R&D team asked by configuring authentication with kerberos to simulate our customer environment and QA be based on this to verify the incoming messages. As we known, Kerberos security for Kafka is optional feature, normal case we don't need to use it in Intranet network or zone.

What's Kerberos

Kerberos is a network authentication protocol. It is designed to provide strong authentication for client/server applications by using secret-key cryptography. A free implementation of this protocol is available from the Massachusetts Institute of Technology. Kerberos is available in many commercial products as well. [ref]

Prerequisite

Kerberos, Kafka and Zookeeper are installed on same host with using same domain. First time don't consider to setup in different hosts with different domain.
A Kerberos realm is the domain over which a Kerberos authentication server has the authority to authenticate a user, host or service.
All hosts must be reachable using hostnames; It is a Kerberos requirement that all your hosts can be resolved with their FQDNs.

Configure

Step 1: Prepare keytab for kafka/client/zookeeper

Using two commands to create principal and export the keytab as file.

sudo /usr/sbin/kadmin.local -q 'addprinc -randkey {principal}/{hostname}@{REALM}'
sudo /usr/sbin/kadmin.local -q "ktadd -k /tmp/keytabs/{keytabname}.keytab kafka/{hostname}@{REALM}"

Kafka

sudo /usr/sbin/kadmin.local -q 'addprinc -randkey kafka/mydomain.com@SUPER_HERO'
sudo /usr/sbin/kadmin.local -q "ktadd -k /tmp/keytabs/kafka.keytab kafka/mydomain.com@SUPER_HERO"

zookeeper

sudo /usr/sbin/kadmin.local -q 'addprinc -randkey zookeeper/mydomain.com@SUPER_HERO'
sudo /usr/sbin/kadmin.local -q "ktadd -k /tmp/keytabs/zookeeper.keytab zookeeper/mydomain.com@SUPER_HERO"

kafka client

sudo /usr/sbin/kadmin.local -q 'addprinc -randkey kafka_client/mydomain.com@SUPER_HERO'
sudo /usr/sbin/kadmin.local -q "ktadd -k /tmp/keytabs/kafka_client.keytab kafka_client/mydomain.com@SUPER_HERO"

Step 2: Kafka and Zookeeper Operation First

Configure server.properties for kafka
vim server.properties

...
#Binding SASL_PLAINTEXT protocol in 9094 port
listeners=PLAINTEXT://localhost:9092,SASL_PLAINTEXT://mydomain.com:9094 
advertised.listeners=PLAINTEXT://localhost:9092,SASL_PLAINTEXT://mydomain.com:9094
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

## inner comuncation with SASL_PLAINTEXT
security.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.kerberos.service.name=kafka
...
zookeeper.connect=mydomain.com:2181
...

Prepare JDK's Kerberos Requirements (/usr/local/etc/kafka/security/krb5.conf)

[libdefaults]
default_realm = SUPER_HERO
forwardable = true
kdc_timeout = 3000
ns_lookup_kdc = false
dns_lookup_realm = false
[realms]
SUPER_HERO = {
  kdc = mydomain.com
  admin_server = mydomain.com
}
[domain_realm]
.mydomain.com = SUPER_HERO
mydomain.com = SUPER_HERO

Step 2.1: Startup Zookeeper

Prepare Zookeeper's Jaas file (/usr/local/etc/kafka/security/zookeeper_jaas.conf)

Server {
  com.sun.security.auth.module.Krb5LoginModule required 
  debug=true
  useKeyTab=true
  keyTab="/usr/local/etc/kafka/security/zookeeper.keytab" <--your zookeper keytab path
  storeKey=true
  useTicketCache=false
  principal="zookeeper/mydomain.com@SUPER_HERO";
};

Export KAFKA_HEAP_OPTS for zookeeper process. MUST ENABLE sun.security.krb5.debug MODE, otherwise it's super hard to find the cause.

export KAFKA_HEAP_OPTS="-Djava.security.krb5.conf=/usr/local/etc/kafka/security/krb5.conf -Djava.security.auth.login.config=/usr/local/etc/kafka/security/zookeeper_jaas.conf  -Dsun.security.krb5.debug=true"

start zookeeper process

zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties

Step 2.2: Startup Kafka

Prepare Kafka's Jaas file (/usr/local/etc/kafka/security/kafka_server_jaas.conf)

KafkaServer {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    debug=true
    serviceName="kafka"
    keyTab="/usr/local/etc/kafka/security/kafka.keytab"
    principal="kafka/mydomain.com@SUPER_HERO";
};

Client {
  com.sun.security.auth.module.Krb5LoginModule required
  debug=true
  useKeyTab=true
  storeKey=true
  keyTab="/usr/local/etc/kafka/security/kafka.keytab"
  principal="kafka/mydomain.com@SUPER_HERO";
};

Export KAFKA_HEAP_OPTS for zookeeper process. MUST ENABLE sun.security.krb5.debug MODE, otherwise it's super hard to find the cause.

export KAFKA_HEAP_OPTS="-Djava.security.krb5.conf=/usr/local/etc/kafka/security/krb5.conf -Djava.security.auth.login.config=/usr/local/etc/kafka/security/kafka_server_jaas.conf -Dsun.security.krb5.debug=true"

start kafka process

kafka-server-start /usr/local/etc/kafka/server.properties

EVERYTHING IS FINE, THE KAFKA LOG LOOK LIKE as

Added key: 16version: 2
Added key: 23version: 2
Added key: 18version: 2
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 18 17 16 23.
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KrbAsReq creating message
>>> KrbKdcReq send: kdc=mydomain.com UDP:88, timeout=3000, number of retries =3, #bytes=281
>>> KDCCommunication: kdc=mydomain.com UDP:88, timeout=3000,Attempt =1, #bytes=281
>>> KrbKdcReq send: #bytes read=815
>>> KdcAccessibility: remove mydomain.com
Looking for keys for: kafka/mydomain.com@SUPER_HERO
Found unsupported keytype (1) for kafka/mydomain.com@SUPER_HERO
Added key: 16version: 2
Added key: 23version: 2
Added key: 18version: 2
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KrbAsRep cons in KrbAsReq.getReply kafka/mydomain.com
principal is kafka/mydomain.com@SUPER_HERO
Will use keytab
Commit Succeeded

[2019-12-21 12:13:43,633] INFO Successfully logged in. (org.apache.kafka.common.security.authenticator.AbstractLogin)
[2019-12-21 12:13:43,634] INFO [Principal=kafka/mydomain.com@SUPER_HERO]: TGT refresh thread started. (org.apache.kafka.common.security.kerberos.KerberosLogin)

Step 2.3: Kafka Client

Prepare client.properties which tell client to use SASL_PLAINTEXT.

security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=kafka
sasl.mechanism=GSSAPI

Prepare Client's Jaas file (/Users/chliu/temp/qa_kafka/jaas.conf)

KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    debug=true
    keyTab="/usr/local/etc/kafka/security/kafka-client.keytab"
    principal="kafka-client/mydomain.com@SUPER_HERO";
};
export KAFKA_OPTS="-Djava.security.auth.login.config=/Users/chliu/temp/qa_kafka/jaas.conf -Djava.security.krb5.conf=/usr/local/etc/kafka/security/krb5.conf -Dsun.security.krb5.debug=true"

Sending message into test topic.

kafka-console-producer  --broker-list mydomain.com:9094 --topic test --producer.config client.properties