Confluent Platform: Connecting Splunk to Kafka

Introduction

The Splunk Source connector provided by Confluent does not support receiving data from a Splunk Universal Forwarder or Splunk Heavy Forwarder.

This connector simply listens on a network port of an application that posts data to Splunk.

In this article, i will be focuing on setting up Splunk Source Connector using a containerized Splunk and Confluent Platform.

For this demonstration, I will be using :

  1. Docker in order to run the Kafka cluster.
  2. Confluent Platform in order to have the necessary commands.

Requirements

First, you need to have docker installed. To install Docker, follow these steps for Ubuntu:

  1. Install Docker Engine
  2. Install Docker Compose
  3. Setup Post-installation steps

Then, you need to install Confluent Platform. It is a fully managed Kafka service and enterprise stream processing platform that allows building Real-time data streaming applications.

After that, add the path of the confluent as follows :

Add the following

source /etc/profile

The Confluent Hub Client is integrated within the Confluent Platform.It allows you to install the different connectors to connect KsqlDB to your data sources.

For this demonstration, we will install the SAP Kafka Source Connector. For the installation, select the second option : 2./(installed rpm/deb package ) . The connectors will be installed in /usr/share/confluent-hub-components .

Confluent Platform along with Splunk using Docker

First, create the /opt/docker-compose directory and put the following docker-compose.yml File.

  • Ensure that the installed connectors are mounted in the connect container inside docker-compose.yml as follows :
  • Ensure that the created certificate are mounted inside the connect containeras follows :

Note that the source connector cannot be created without providing in splunk.ssh.key.store.path even though SSL is disabled, for that reasons i have created a random key store and have used it just for my connector to be created, it is not important as SSL is disabled, things gonna work.

Fore more information about key store check out this article on Confluent Security

inside the security folder, Create a key store named kafka1 run the following

This will give you :

  • Ensure that a volume is associated with the connect containeras follows :

Demonstration

First, run docker-compose:

Configuring Splunk HTTP Event Collector (HEC) token

Access to Splunk’s UI

Once connected , click on the Splunk Enterprise icon and change your license group to Free license. Then, save and restart Splunk.

Click on Settings then on Data inputs

Then choose the HTTP Event Collector option

Click on Global Settings and put the following configuration

Then save. Next, create a new token as follows:

Submit it :

One done, copy your HEC token somewhere.

Creating Splunk Source Connector

  1. Access your confluent control-center :

2.In the connect part, click on upload connector then upload the following connector

You can check connector’s logs as follows

4. Put some data to Splunk using your HEC token

You will see that your data have been successfully posted to Splunk when you search it based on your HEC.

However, no data have been imported to the splunk-source topic.

This is due to the fact that The Splunk Source connector provided by Confluent does not support receiving data from a Splunk Universal Forwarder or Splunk Heavy Forwarder. It imply listens on a network port of an application that posts data to Splunk.

So, because our connector resides in the Connect container and listens on port 8889 . Lets connect to that container and post some data :

Now, when you run :

You can see that the posted data have been imported, this is due to the fact that in the second POST we didn’t deal with Splunk https://splunk:8889, we have imitated Splunkhttps://localhost:8889

You can run the following commands to stop and clean docker

References

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store