Convert Slack into a Reporting Tool with AWS

With analytics taking over every aspect of our lives, it has now become a mandate for data engineers like us to ensure that the insights are always available on the go.

Be it a financial project where transactions are happening every minute, or a key performance metrics project where values are judged to power important process changes and decisions, the availability of the numbers have to be correct, easy and instant.

Picture Credit: https://slackhq.com/aws-chatbot-bring-aws-into-your-slack-channel

With world around us moving at warp speed, we communicate with each other mostly using messenger applications like Slack, Telegram or Whatsapp since it undeniably runs at par in terms of speed with our lives and the information retrieval and transfer is instant.

Today, we will try to achieve the same efficiency and flexibility for our data insights by converting Slack into a reporting tool.

Before we start, I would like to highlight that this tutorial requires an intermediate knowledge of AWS Services, Slack API and Python programming. While, I would try my best to keep it simple, and provide necessary reference links, please do some referential reading and reach out to me if there are questions.

Requirements

If you are a part of an enterprise group which uses Slack and AWS or you just want to take a stab at it because it sounds cool, this implementation is nice and quite easy. Here are the basic requirements before we move on to the details:

• A Slack Account with a slack app installed: To get started with slack, visit www.slack.com and for steps regarding installing a slack app, visit api.slack.com/apps.
• S3: A simple storage service (S3!) provided by AWS that would store our data
• AWS Lambda: AWS Lambda is a server-less code running service. We would require a Python based Lambda Function on AWS
• AWS Gateway: This would provide an HTTP API link that would accept requests from slack, trigger the Lambda based function and send the output as a response

For people trying this implementation for practice or fun, the AWS free tier is more than enough for this implementation. However, based on your usage, I would request you to check your requirement compared to the AWS free tier at https://aws.amazon.com/free/ to avoid bill shocks.

Design Flow

Here, I will assume that required data for the implementation is already available. For this implementation, I have got my data on a CSV file placed in an S3 Bucket.

Following is the design diagram with required numerical markings for steps. Each step has been explained below the diagram.

High Level Design Diagram

1. The user sends a request using the slack slash command to the API Gateway URL. After an app is created, slash command is added to its interface for additional interaction capabilities. The steps to add a URL to slack slash command to send the request packet to is covered in this article later
2. API Gateway sends the request packet towards the AWS Lambda Function and triggers it to process it
3. AWS Lambda function parses the request packet and converts it from its base64 encoded format (that’s the default format slack sends) into textual format. From the request packet text, the function converts it into a dictionary and check for two conditions — Slack Token and Enterprise Name. If both the conditions are fulfilled, Lambda then processes the request to gather a response. Our data for this implementation resides on S3 and a set of steps are triggered to gather relevant data from S3 and process it using Pandas
4. Based on the Lambda function, S3 returns the data from its bucket and the Lambda function parses it to easy slack message format
5. Lambda function returns a response which is passed to the API Gateway. The response is preformatted from step 4 based on a needed messaging template. For example, if you want to send out top 15 salesmen and their statistics to the sales manager, the messaging format template should be created in a slack friendly way to show top 15 salesmen, their rank and total sales for a given week
6. Finally the API Gateway sends the response back to the slack command and the user sees the response under his/her chat window

K. This Step is an additional step which keeps our lambda function alive. The thing about lambda functions is that whenever they are triggered, AWS collects the needed resources to run them, unless of-course they are reserved for them permanently. The time needed to gather resources cause timeout error on slack as slack expects a response within 3 seconds. To counter that we have this additional step. While my solution is not apt for all the scenarios, here, I have created another Lambda function called the Lambda warmer which triggers our lambda function every 12 minutes to prevent it from going cold. This keeps the function alive with the needed sources, without me having to pay to reserve resources for it.

L. This is just the representation of all the logging we do from the lambda function to cloudwatch

Implementation

I know it will be a bit overwhelming for the new developers to go through so much details and understand the keywords and nuances of each step. However, I want to ensure that my solution reaches out to maximum engineers of all experience levels so here I provide you with the sourcecode of the whole Lambda function and a bit more!

To get the source code, clone the repository from here: https://github.com/abhikchakraborty92/aws-lambda-slack-slash-template.git

Now, let’s talk about how it works:

Lambda Function Setup

• Fire up a blank Lambda function and copy the code under ‘main.py’ to it. This function is our main Lambda function which would perform authentication, read the slack command and related parameters, process the response and send it to slack
• After adding the code, add a AWS API Gateway as a trigger to the lambda function. Basically, this will provide an HTTP URL which will receive GET requests from the slack server and respond according to the response formulated by the Lambda function

API Gateway Details. Here the API endpoint is the HTTP URL

• Go to IAM under AWS services and create a new Lambda role. This role should have permissions to execute Lambda function, write logs on cloudwatch, and read S3 files. We will use this role both for our Lambda Warmer function and the main function
• Create a slack app by going to api.slack.com/apps. Open your app interface and under ‘Basic Information’, get your slack verification token under App Credentials. Add this token under environment variables of the Lambda function. There is an optional step of encrypting the credential too but I will not be covering it here

App Verification Token

I know it says it is a deprecated feature but I have been seeing it for last one year. Once this is gone, I will share an updated code with new steps for verification

Code Sections

The whole code under main.py is self sufficient with only minute changes required based on the requirement. I have added necessary comments in the code itself. Let’s walk through different functions under it and their use.

Getdata()

This implementation fetches the data directly from a CSV file in the S3 bucket saved under the variable bucket and stored under the key — file_key. Just replace the current values of the variable with the relevant value from your S3 bucket and File key where your file is stored.

request_parse(event)

This particular function will be used to decode the base64 encoded slack event body. The event body is a mandatory parameter in the lambda handler function. This function is invoked from the dispatcher function (below) and other functions in this implementation to gather different components of the request body like verification token, enterprise name, response URL, etc. Using that, we can run various other functions in this code.

verify(event)

This function uses the contents of the request body obtained from the request_parse function to verify if the request is coming from a valid slack based point of source. The conditions checked are: — Validity of slack verification token and Enterprise Name. For free slack users, the enterprise name verification is an optional step but I would highly recommend having Enterprise verification for enterprise users.

api_handler(event)

One of the most important functions of this template is the API handler. Here we add the routes (slack slash commands) and related functions or steps to be triggered whenever that route is called. Here, based on the slash command, the getdata() function is called to get the data from the S3 bucket and slack_message_generator() function is called to parse the data into slack readable format based on a given search string. Before doing that, this function is responsible for sending an initial response to slack to ensure that the timeout error caused due to no response within 3 seconds is avoided. Finally, it also handles conditions where the slack slash command is not correct and sends a default error message.

slack_message_generator(search_string,dataframe)

This function accepts search_string and dataframe as two parameters. Search String is the string sent along with the slash command to filter relevant data. For example, we have daily sale figures for all the salesmen in the S3 bucket and the requester from slack wants to view the sales figure of a particular salesman. In this case, the search_string can be the name of the salesman or the ID of the salesman. Dataframe is the tabular data obtained using the getdata() function. This is where the search_string would be used to filter relevant data and the filtered data is parsed into readable message format in this function. This function also handles the case if the search_string is empty or if the search_string returned no values

dispatcher(event,context)

This function is basically the trigger point of the whole implementation. Based on the slash command, it triggers other functions for verification and might reject a request if it is not from an authentic slack application. This is the only function triggered from the event handler.

You can totally ignore this function and handle the request directly from the lambda_handler. But, to avoid confusion for myself, I kept this function as a separate one.

lambda_handler(event,context)

If you are aware of the lambda functions, this function is available by default whenever we create a new python lambda implementation. The purpose of this is to have a set of steps and return results from the lambda. However, we don’t have much use of it here as we have defined definitive functions for specific tasks. We will use this function just to call the dispatcher

Lambda Warmer

This is a very simple function which is already written under lambda_warmer.py. This function can be used as a single implementation to warm all your lambda functions.

• To use this open a lambda function on AWS and copy the code and the file function_list.csv
• In the file function_list.csv, type the name of the function which you want to warm and the ARN link for it. In our case, it will be our main lambda function

ARN Link Location on a Lambda Function

• Add a AWS EventBridge (cloudwatch event) as a trigger for this function to trigger it every n minutes. You can use CRON expression or Rate Expression to do it. For me, I have set n at 12 minutes to run the function every 12 minutes

EventBridge Description for the Lambda Warmer

Slack App Setup

After the AWS setup is over and you are done with the implementation, it is time for the slack app to be ready to send relevant requests to your AWS API link. To do that, we have to set our slack app with some slash commands.

• A slash command is nothing but an expression to tell slack that some request has to be sent to a particular URL to gather relevant response. This normally starts with a slash (/)
• To set up a slash command, fire up a slack application and then on the app home page, click on Slash commands on the left panel under Features
• A page will open with your current slash commands if you have any. If not, press the button ‘Create new Command’ to open the slash command window
• The next step is simple. Under Command, type the word you want the user to type to trigger. For example, /salesmen-insight can be used as a command for getting data regarding salesmen. In the next section, under Request URL, type the AWS API Gateway URL that was generated during the lambda function setup. Finally, add a Short Description and a Usage Hint to make it easy for the users to use the command.

Slash command window

• It’s done! For more information and queries, refer to the official documentation from slack here

Testing your Implementation

Configure test event under your AWS Lambda function to test your implementation. To do that, add a key under your test event with the name ‘body’ and add a base64 encoded value of the request body. Normally, the request body has the following structure:

Add relevant values to the request body above and convert it into base64 encoded format. Add it as a value to the ‘body’ key and save the test event. That’s it. You can now use the test event to test your implementation.

That should do it. We have created the implementation that would bring the insights right into people’s fingertips. No more going through dashboards, searching for that weekly status email, or randomly waking up a colleague in another timezone to ask for that adhoc insight.

Slack just became a reporting tool!