Serverless logging with Amazon OpenSearch Serverless and Amazon Kinesis Information Firehose

On this put up, you’ll find out how you should use Amazon Kinesis Information Firehose to construct a log ingestion pipeline to ship VPC circulate logs to Amazon OpenSearch Serverless. First, you create the OpenSearch Serverless assortment you employ to retailer VPC circulate logs, then you definately create a Kinesis Information Firehose supply pipeline that forwards the circulate logs to OpenSearch Serverless. Lastly, you allow supply of VPC circulate logs to your Firehose supply stream. The next diagram illustrates the answer workflow.

OpenSearch Serverless is a brand new serverless possibility supplied by Amazon OpenSearch Service. OpenSearch Serverless makes it easy to run petabyte-scale search and analytics workloads with out having to configure, handle, or scale OpenSearch clusters. OpenSearch Serverless mechanically provisions and scales the underlying assets to ship quick information ingestion and question responses for even probably the most demanding and unpredictable workloads.

Kinesis Information Firehose is a well-liked service that delivers streaming information from over 20 AWS companies to over 15 analytical and observability instruments comparable to OpenSearch Serverless. Kinesis Information Firehose is nice for these in search of a quick and simple solution to ship your VPC circulate logs information to your OpenSearch Serverless assortment in minutes with no single line of code and with out constructing or managing your personal information ingestion and supply infrastructure.

VPC circulate logs seize the site visitors info going to and out of your community interfaces in your VPC. With the launch of Kinesis Information Firehose assist to OpenSearch Serverless, it makes a simple answer to investigate your VPC circulate logs with only a few clicks. Kinesis Information Firehose offers a real end-to-end serverless mechanism to ship your circulate logs to OpenSearch Serverless, the place you should use OpenSearch Dashboards to go looking by way of these logs, create dashboards, detect anomalies, and ship alerts. VPC circulate logs lets you reply questions like:

• What proportion of your site visitors is getting dropped?
• How a lot site visitors is getting generated for particular sources and locations?

Create your OpenSearch Serverless assortment

To get began, you first create a group. An OpenSearch Serverless assortment is a logical grouping of a number of indexes that signify an analytics workload. Full the next steps:

1. On the OpenSearch Service console, select Collections below Serverless within the navigation pane.
2. Select Create a group.
3. For Assortment title, enter a reputation (for instance, vpc-flow-logs).
4. For Assortment kind¸ select Time sequence.
5. For Encryption, select your most well-liked encryption setting:
   1. Select Use AWS owned key to make use of an AWS managed key.
   2. Select a special AWS KMS key to make use of your personal AWS Key Administration Service (AWS KMS) key.
6. For Community entry settings, select your most well-liked setting:
   1. Select VPC to make use of a VPC endpoint.
   2. Select Public to make use of a public endpoint.

AWS recommends that you just use a VPC endpoint for all manufacturing workloads. For this walkthrough, choose Public.

6. Select Create.

It ought to take couple of minutes to create the gathering.

The next graphic offers a fast demonstration of making the OpenSearch Serverless assortment by way of the previous steps.

At this level, you’ve gotten efficiently created a group for OpenSearch Serverless. Subsequent, you create a supply pipeline for Kinesis Information Firehose.

Create a Kinesis Information Firehose supply stream

To arrange a supply stream for Kinesis Information Firehose, full the next steps:

1. On the Kinesis Information Firehose console, select Create supply stream.
2. For Supply, specify Direct PUT.

Try Supply, Vacation spot, and Identify to be taught extra about totally different sources supported by Kinesis Information Firehose.

3. For Vacation spot, select Amazon OpenSearch Serverless.
4. For Supply stream title, enter a reputation (for instance, vpc-flow-logs).
5. Below Vacation spot settings, within the OpenSearch Serverless assortment settings, select Browse.
6. Choose vpc-flow-logs.
7. Select Select.

In case your assortment continues to be creating, wait a couple of minutes and take a look at once more.

8. For Index, specify vpc-flow-logs.
9. Within the Backup settings part, choose Failed information solely for the Supply document backup in Amazon S3.

Kinesis Information Firehose makes use of Amazon Easy Storage Service (Amazon S3) to again up failed information that it makes an attempt to ship to your chosen vacation spot. If you wish to maintain all information, choose All information.

10. For S3 Backup Bucket, select Browse to pick out an current S3 bucket, or select Create to create a brand new bucket.
11. Select Create supply stream.

The next graphic offers a fast demonstration of making the Kinesis Information Firehose supply stream by way of the previous steps.

At this level, you’ve gotten efficiently created a supply stream for Kinesis Information Firehose, which you’ll use to stream information out of your VPC circulate logs and ship it to your OpenSearch Serverless assortment.

Arrange the info entry coverage in your OpenSearch Serverless assortment

Earlier than you ship any logs to OpenSearch Serverless, it’s essential create a knowledge entry coverage inside OpenSearch Serverless that permits Kinesis Information Firehose to put in writing to the vpc-flow-logs index in your assortment. Full the next steps:

1. On the Kinesis Information Firehose console, select the Configuration tab on the main points web page for the vpc-flow-logs supply stream you simply created.
2. Within the Permissions part, notice down the AWS Id and Entry Administration (IAM) function.
3. Navigate to the vpc-flow-logs assortment particulars web page on the OpenSearch Serverless dashboard.
4. Below Information entry, select Handle information entry.
5. Select Create entry coverage.
6. Within the Identify and outline part, specify an entry coverage title, add an outline, and choose JSON because the coverage definition methodology.
7. Add the next coverage within the JSON editor. Present the gathering title and index you specified throughout the supply stream creation within the coverage. Present the IAM function title that you just bought from the permissions web page of the Firehose supply stream, and the account ID in your AWS account.

   [
     {
       "Rules": [
         {
           "ResourceType": "index",
           "Resource": [
             "index/<collection-name>/<index-name>"
           ],
           "Permission": [
             "aoss:WriteDocument",
             "aoss:CreateIndex",
             "aoss:UpdateIndex"
           ]
         }
       ],
       "Principal": [
         "arn:aws:sts::<aws-account-id>:assumed-role/<IAM-role-name>/*"
       ]
     }
   ]

8. Select Create.

The next graphic offers a fast demonstration of making the info entry coverage by way of the previous steps.

Arrange VPC circulate logs

Within the ultimate step of this put up, you allow circulate logs in your VPC with the vacation spot as Kinesis Information Firehose, which sends the info to OpenSearch Serverless.

1. Navigate to the AWS Administration Console.
2. Seek for “VPC” after which select Your VPCs within the search consequence (hover over the VPC rectangle to disclose the hyperlink).
3. Select the VPC ID hyperlink for considered one of your VPCs.
4. On the Circulate Logs tab, select Create circulate log.
5. For Identify, enter a reputation.
6. Go away the Filter set to All. You possibly can restrict the site visitors by deciding on Settle for or Reject.
7. Below Vacation spot, choose Ship to Kinesis Firehose in the identical account.
8. For Kinesis Firehose supply stream title, select vpc-flow-logs.
9. Select Create circulate log.

The next graphic offers a fast demonstration of making a circulate log in your VPC following the previous steps.

Study the VPC circulate logs information in your assortment utilizing OpenSearch Dashboards

You gained’t be capable to entry your assortment information till you configure information entry. Information entry insurance policies enable customers to entry the precise information inside a group.

To create a knowledge entry coverage for OpenSearch Dashboards, full the next steps:

1. Navigate to the vpc-flow-logs assortment particulars web page on the OpenSearch Serverless dashboard.
2. Below Information entry, select Handle information entry.
3. Select Create entry coverage.
4. Within the Identify and outline part, specify an entry coverage title, add an outline, and choose JSON because the coverage definition methodology.
5. Add the next coverage within the JSON editor. Present the gathering title and index you specified throughout the supply stream creation within the coverage. Moreover, present the IAM person and the account ID in your AWS account. It is advisable to just remember to have the AWS entry and secret keys for the principal that you just specified as an IAM person.

   [
     {
       "Rules": [
         {
           "Resource": [
             "index/<collection-name>/<index-name>"
           ],
           "Permission": [
             "aoss:ReadDocument"
           ],
           "ResourceType": "index"
         }
       ],
       "Principal": [
         "arn:aws:iam::<aws-account-id>:user/<IAM-user-name>"
       ]
     }
   ]

6. Select Create.
7. Navigate to OpenSearch Serverless and select the gathering you created (vpc-flow-logs).
8. Select the OpenSearch Dashboards URL and log in along with your IAM entry key and secret key for the person you specified below Principal.
9. Navigate to dev instruments inside OpenSearch Dashboards and run the next question to retrieve the VPC circulate logs in your VPC:

   GET <index-name>/_search
   {
     "question": {
       "match_all": {}
     }
   }

The question returns the info as proven within the following screenshot, which incorporates info comparable to account ID, interface ID, supply IP deal with, vacation spot IP deal with, and extra.

Create dashboards

After the info is flowing into OpenSearch Serverless, you possibly can simply create dashboards to observe the exercise in your VPC. The next instance dashboard exhibits general site visitors, accepted and rejected site visitors, bytes transmitted, and a few charts with the highest sources and locations.

Clear up

In the event you don’t need to proceed utilizing the answer, you should definitely delete the assets you created:

1. Return to the AWS console and within the VPCs part, disable the circulate logs in your VPC.
2. Within the OpenSearch Serverless dashboard, delete your vpc-flow-logs assortment.
3. On the Kinesis Information Firehose console, delete your vpc-flow-logs supply stream.

Conclusion

On this put up, you created an end-to-end serverless pipeline to ship your VPC circulate logs to OpenSearch Serverless utilizing Kinesis Information Firehose. On this instance, you constructed a supply pipeline in your VPC circulate logs, however you too can use Kinesis Information Firehose to ship logs from Amazon Kinesis Information Streams and Amazon CloudWatch, which in flip might be despatched to OpenSearch Serverless collections for working analytics on these logs. With serverless options on AWS, you possibly can focus in your utility improvement moderately than worrying concerning the ingestion pipeline and instruments to visualise your logs.

Get hands-on with OpenSearch Serverless by taking the Getting Began with Amazon OpenSearch Serverless workshop and take a look at different pipelines for analyzing your logs.

You probably have suggestions about this put up, share it within the feedback part. You probably have questions on this put up, begin a brand new thread on the Amazon OpenSearch Service discussion board or contact AWS Assist.

---

Concerning the authors

Jon Handler (@_searchgeek) is a Principal Options Architect at Amazon Net Providers primarily based in Palo Alto, CA. Jon works intently with the CloudSearch and Elasticsearch groups, offering assist and steerage to a broad vary of consumers who’ve search workloads that they need to transfer to the AWS Cloud. Previous to becoming a member of AWS, Jon’s profession as a software program developer included 4 years of coding a large-scale, eCommerce search engine.

Prashant Agrawal is a Sr. Search Specialist Options Architect with Amazon OpenSearch Service. He works intently with prospects to assist them migrate their workloads to the cloud and helps current prospects fine-tune their clusters to attain higher efficiency and save on value. Earlier than becoming a member of AWS, he helped numerous prospects use OpenSearch and Elasticsearch for his or her search and log analytics use circumstances. When not working, you’ll find him touring and exploring new locations. In brief, he likes doing Eat → Journey → Repeat.