Monitor AWS workloads with no single line of code with Logz.io and Kinesis Firehose

Observability knowledge gives close to real-time insights into the well being and efficiency of AWS workloads, in order that engineers can rapidly tackle manufacturing points and troubleshoot them earlier than widespread buyer affect.

As AWS workloads develop, observability knowledge has been exploding, which requires versatile large knowledge options to deal with the throughput of huge and unpredictable volumes of observability knowledge.

Answer overview

One choice is Amazon Kinesis Knowledge Firehose, which is a well-liked service for streaming large volumes of AWS knowledge for storage and analytics. By pulling knowledge from Amazon CloudWatch, Amazon Kinesis Knowledge Firehose can ship knowledge to observability options.

Amongst these observability options is Logz.io, which might now ingest metric knowledge from Amazon Kinesis Knowledge Firehose and make it simpler to get metrics out of your AWS account to your Logz.io account for evaluation, alerting, and correlation with logs and traces.

In just a few clicks and some configurations, we’ll see how one can begin streaming your metric knowledge (and shortly, log knowledge!) to Logz.io for storage and evaluation.

Stipulations

Logz.io account – Create a free trial right here
Logz.io transport token – Study metrics tokens right here. It’s good to be a Logz.io administrator.
Entry to Amazon CloudWatch and Amazon Kinesis Knowledge Firehose with the suitable permissions to handle HTTP endpoints.
Acceptable permissions to create an Amazon Easy Storage Service (Amazon S3) bucket

Sending Amazon CloudWatch metric knowledge to Logz.io with an Amazon Kinesis Knowledge Firehose

Amazon Kinesis Knowledge Firehose is a service for ingesting, processing, and loading knowledge from giant, distributed sources comparable to logs or clickstreams into a number of shoppers for storage and real-time analytics. Kinesis Knowledge Firehose helps greater than 50 sources and locations as of right this moment. This integration might be arrange in minutes with no single line of code and allows close to real-time analytics for observability knowledge generated by AWS providers through the use of Amazon CloudWatch, Amazon Kinesis Knowledge Firehose, and Logz.io.

As soon as the combination is configured, Logz.io prospects can open the Infrastructure Monitoring product to see their knowledge coming in and populating their dashboards. To see among the knowledge analytics and correlation you get with Logz.io, take a look at this brief demonstration.

Let’s start a step-by-step tutorial for organising the combination.

Begin by going to Amazon Kinesis Knowledge Firehose and making a supply stream with Knowledge Firehose.

Subsequent you choose a supply and vacation spot. Choose Direct Put because the supply and Logz.io the vacation spot.
Subsequent, configure the vacation spot settings. Give the HTTP endpoint a reputation, which ought to embody logz.io.
Choose from the dropdown the suitable endpoint you wish to use.

In the event you’re sending knowledge to a European area, then set it to Logz.io Metrics EU. Or you should use the us-east-1 vacation spot by deciding on Logz.io Metrics US.

Subsequent, add your Logz.io Transport Token. You will discover this by going to Settings in Logz.io and deciding on Handle Tokens, which requires Logz.io administrator to entry. This ensures that your account is simply ingesting knowledge from the outlined sources (e.g., this Amazon Kinesis Knowledge Firehose supply stream).

Maintain Content material encoding on Disabled and set your required Retry Period.

You may also configure Buffer hints to your preferences.

Subsequent, decide your Backup settings in case one thing goes fallacious. Generally, it’s solely essential to again up the failed knowledge. Merely select an Amazon S3 bucket or create a brand new one to retailer knowledge if it doesn’t make it to Logz.io. Then, choose Create a supply stream.

Now it’s time to attach Amazon CloudWatch to our Amazon Kinesis Knowledge Firehose Supply Stream.

Navigate to Amazon CloudWatch and choose Streams within the Metrics menu. Choose Create metrics stream.
Subsequent, you possibly can both choose to ship all of your Amazon CloudWatch metrics to Logz.io, or solely metrics from specified namespaces.

On this case, we selected Amazon Elastic Compute Cloud (Amazon EC2), Amazon Relational Database Service (Amazon RDS), AWS Lambda, and Elastic Load Balancing (ELB).

Underneath Configuration, select the Choose an current Firehose owned by your account choice and select the Amazon Kinesis Knowledge Firehose you simply configured.

In the event you’d like, you possibly can select extra statistics within the Add extra statistics field, which gives useful metrics when it comes to percentiles to observe like latency metrics (i.e., which providers have the best common latency). This will enhance your prices.

Lastly, give your metric stream a reputation and hit Create metric stream.

That’s it! With out writing a single line of code, we configured an integration with AWS and Logz.io that allows quick and simple infrastructure monitoring by means of Amazon CloudWatch knowledge assortment.

Your metrics shall be saved in Logz.io for 18 months out of the field, with out requiring any overhead administration.

You may also start to construct dashboards and alerts to start monitoring – like this Amazon EC2 monitoring dashboard beneath.

Conclusion

This put up demonstrated how one can configure an integration with AWS and Logz.io for environment friendly infrastructure monitoring by means of Amazon CloudWatch.

To be taught extra about constructing metrics dashboards in Logz.io, you possibly can watch this video.

At the moment, some customers would possibly discover that they’re sending extra knowledge than they really want, which might elevate prices. In future variations of this integration, will probably be simpler to slender down the metrics to cut back prices.

Wish to attempt it your self? Create a Logz.io account right this moment, navigate to our infrastructure monitoring product, and begin streaming metric knowledge to Logz.io to begin monitoring.