Deploying Functionbeat using Azure DevOps Release Pipelines

Functionbeat Architecture

Elasticsearch is a trending search engine option where you can store various types of data such as application logs, events, time series, etc... and then perform search and analytics on the data in a scalable fast manner.

Data can be fed into Elasticsearch using various tools such as Logstash and Beats which have different types for different scenarios, our focus is the serverless type which is called Functionbeat.

What is Functionbeat:

Functionbeat is a Beat that you deploy as a function in your serverless environment to collect data from cloud services and ship it to the Elastic Stack.

How to Implement CD for your Functionbeat:

As an example, we will work on an AWS SQS Functionbeat with the following functionbeat.yml configurations.

First, we need to add a new release pipeline by following the steps shown below.

Create a new release pipeline in Azure DevOps

Then choosing Empty Job and specifying a name, then clicking on save.

Note: I will be working with Functionbeat v7.10.2 (latest available at the date of this post)

1- Preparing needed dependencies:

In order to deploy Functionbeat we need to download Functionbeat package, which contains around 130MB worth of files, so it will be a bad idea to add the whole package to your Git repository.

All you need to push to your repository is the few files that contain your configurations (mainly functionbeat.yml), and you can download the package each time you create a new release by adding a new Command Line task with the following script that downloads and unzips Functionbeat package.

curl -L -O https://artifacts.elastic.co/downloads/beats/functionbeat/functionbeat-7.10.2-linux-x86_64.tar.gz
tar xzvf functionbeat-7.10.2-linux-x86_64.tar.gz

Now, to write shorter commands let’s change the directory to Functionbeat package directory

cd functionbeat-7.10.2-linux-x86_64

Azure DevOps Download Functionbeat

2- Deployment:

To deploy to AWS cloud from Azure DevOps you need to authorize Azure DevOps agents by adding AWS credentials with the needed permissions.

We need to provide 3 environment variables related to AWS access in order for Functionbeat deploy command to work.

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION

Also instead of storing sensitive information in functionbeat.yml such as:

event_source_arn
cloud.id
cloud.auth

We will pass them as environment variables, for example:

cloud.id: ${ES_Cloud_Id:?You must set ES_Cloud_Id environment var}

The above line will pass the value of environment variable ES_Cloud_Id to cloud.id and will output a warning if ES_Cloud_Id is not set.

I will use secret variables to store sensitive information like:

Azure DevOps secret variables to deploy Functionbeat

For security, make sure to lock the variables holding sensitive information to turn them into secrets.

To make your secrets available as Environment variables go to the Command Line task that you have already created and add the following environment variables.

Command Line Task Environment Variables

Now that our credentials are secure and ready, we can add deploy command to our same Command Line Task.

To create the Functionbeat lambda you need to run.

./functionbeat -v -e -d "*" deploy {function name as in configs} -c {path to functionbeat.yml}

which will work successfully only for the first release, and you will need to run the following later then.

./functionbeat -v -e -d "*" update {function name as in configs} -c {path to functionbeat.yml}

So I suggest you combine both like:

./functionbeat -v -e -d "*" update {function name as in configs} -c {path to functionbeat.yml} || ./functionbeat -v -e -d "*" deploy {function name as in configs} -c {path to functionbeat.yml}

Which will run deploy only if the update fails.

Note that update comes first because all releases will update your Functionbeat Lambda while only the first release will create it.

In my case, my command looks like this:

./functionbeat -v -e -d "*" update sqsbeats -c ../functionbeat.yml|| ./functionbeat -v -e -d "*" deploy sqsbeats -c ../functionbeat.yml

The whole Command Line script to Deploy Functionbeat.

echo "Downloading Functionbeat package and unzip"
curl -L -O https://artifacts.elastic.co/downloads/beats/functionbeat/functionbeat-7.10.2-linux-x86_64.tar.gz
tar xzvf functionbeat-7.10.2-linux-x86_64.tar.gz
echo "Changing directory to Functionbeat package files"
cd functionbeat-7.10.2-linux-x86_64
echo "Deploying Functionbeat to AWS"
./functionbeat -v -e -d "*" update sqsbeats -c ../functionbeat.yml|| ./functionbeat -v -e -d "*" deploy sqsbeats -c ../functionbeat.yml

Deploy Functionbeat to AWS using DevOps Command Line Task

Now you can run your release and as a result, you should see a new stack created in your AWS Cloudformation with the following resources:

Functionbeat AWS resources

Links:

Step 4: Deploy Functionbeat code to your serverless environment | Functionbeat Reference [6.8] |…

Functionbeat command reference | Functionbeat Reference [7.10] | Elastic