SolarWinds Observability and AWS Storage

pavel.hardak 1 month ago 8 minute read time

Amazon Web Services^® (AWS) offers multiple data storage services, including:

S3^® for object storage
Elastic Block Store (EBS) for block storage
Elastic File System (EFS) for file storage

As organizations aim to optimize their costs by leveraging the most economical storage solutions, they lean on metrics to track storage usage. AWS provides some basic monitoring and metrics with CloudWatch^®, but SolarWinds^® Observability can offer deeper insights with more granular data, historical analytics, and predictive models.

SolarWinds Observability offers a robust suite of tools that provides comprehensive visibility of your infrastructure—including AWS cloud resources—for easy monitoring, troubleshooting, and optimization. Using the platform will help you ensure the availability, performance, reliability, and security of data stored on AWS.

This article will explore the different AWS storage options and the challenges associated with monitoring those solutions. Then, we’ll look at how to monitor these services more effectively with SolarWinds Observability.

A Brief Rundown of AWS Cloud Storage Options

Depending on your needs, AWS offers different storage options organized by data type: file, block, and object. Understanding how AWS storage options work is essential, as your application's storage choice directly impacts performance, scalability, and cost-effectiveness.

Storage options offered by Amazon Web Services.

EFS for file storage

EFS offers serverless and dynamically scalable file storage, eliminating the need to manage storage capacity and performance provisioning. It automatically grows or shrinks in size as you add and remove files. EFS integrates seamlessly with other AWS services, including EC2^®, EKS^®, and Lambda^®.

EFS volumes can be mounted to multiple devices, facilitating multi-AZ support for enhanced redundancy. Regarding performance, the disk latency per I/O operation for EFS is higher (worse) than that of EBS.

EFS supports multiple use cases, including storage for:

Containerized and serverless apps
Big data analytics
Web serving
Content management
Application development
Media workflows
Database backups

EBS for block-level storage

EBS offers block-level storage volumes that work seamlessly with EC2 instances. It offers persistent and low-latency storage optimized for transactional workloads. Unlike EFS, EBS volumes are accessed at the block level. They are typically used for applications that require consistent and high-performance storage, such as databases and transactional systems.

S3 for object storage

S3 is an object storage service with high durability, scalability, security, and performance. S3 offers features like versioning, encryption, and life cycle management.

S3 stores data as objects within buckets and is optimized for storing vast amounts of unstructured data, such as images, videos, and documents. It’s suitable for use cases that include:

Backup and restore
Content distribution
Data archiving
Serving static assets for websites and applications

Selecting an AWS storage option

Choosing the appropriate storage service is essential to ensure your application meets performance expectations. For example, if your application requires storing and retrieving large media files, S3 would be the preferred choice due to its scalability and low cost. On the other hand, if your application demands high-performance block-level storage with low latency, EBS would be more suitable for hosting databases or applications that require persistent storage.

Using the wrong storage option for your application can result in performance issues, cost inefficiencies, or scalability challenges. For example, a content management system (CMS) using EFS to store images and documents could lead to high costs as the user base grows. In that scenario, S3 with CloudFront^®would be faster and cheaper for serving static assets globally.

Challenges with AWS Storage Monitoring

At a bare minimum, organizations need good visibility into the usage and performance of their AWS storage solutions. This visibility provides foundational information for capacity planning, cost optimization, and decision-making. However, storage monitoring in AWS brings some key challenges.

Clear visibility

Understanding how storage resources are utilized and whether they meet your application’s performance requirements is essential. Some organizations find it difficult to set up or find proper monitoring of these metrics in AWS. Without easy access to reliable storage metrics—such as latency, error rates, and cost—it's difficult to proactively address the application’s performance issues.

Predicting future storage needs

For some applications, it is difficult to anticipate how storage usage will evolve. For example, as the volume of ingested data increases, the cost of an EC2 instance using EFS volumes for big data analytics can grow rapidly. However, continuously monitoring storage metrics—such as the amount of data stored, throughput, and data access—makes predicting storage costs easier.

Data security and compliance

It is crucial to maintain data security and compliance when storing sensitive data. For example, consider an application where EBS is used with EC2 instances to host databases containing customer information. These databases must comply with data protection regulations (such as GDPR or CCPA), requiring you to implement security measures that include encryption, access controls, and audits. Monitoring storage-related metrics, such as encryption status, access logs, and audit trails, is essential to safeguard sensitive data.

Monitoring AWS Storage with SolarWinds Observability

AWS cloud platform monitoring from SolarWinds Observability provides comprehensive insights into your AWS cloud resources by collecting data about your AWS services' performance, stability, and overall health.

You can help ensure your applications use the most optimal storage option by continuously tracking storage metrics. Proactive monitoring helps you take action by modifying the existing storage service or swapping it out with a different one to ensure cost-effectiveness, availability, performance, reliability, and security.

You can use AWS CloudWatch^® to monitor storage resources and applications in your infrastructure, but the solution doesn't provide tools to analyze real-time and long-term trends in your system. In contrast, AWS cloud platform monitoring from SolarWinds Observability provides deeper, actionable insights for optimizing performance and cost.

SolarWinds Observability offers native integration for all AWS storage services. In this section, we’ll guide you through integrating SolarWinds Observability to monitor AWS storage services so that you can gain insights into usage, cost, and performance.

Adding AWS data in SolarWinds Observability

Integrating SolarWinds Observability for AWS Storage Monitoring

Follow these steps to start monitoring your AWS storage services:

Add an AWS cloud account to the SolarWinds Observability platform, either manually by using the UI console or by using AWS CloudFormation.
Follow this guide for integration instructions based on your preferences.

Once you connect your AWS account with SolarWinds Observability, it will start gathering metrics for the enabled services. Follow this guide to enable detailed CloudWatch metrics for your storage services to gather additional metrics.

Be sure to note the following:

For EBS, you must ensure that CloudWatch metrics are enabled on the AWS console.

For EFS, you must ensure that the EFS volume is created in general-purpose mode to collect additional metrics.
For S3, you must enable S3 request metrics in CloudWatch to collect additional metrics.

Metrics collected from these storage services are displayed as widgets on the Explore page of SolarWinds Observability.

Additional metrics are available in the Metrics Explorer. You can search for AWS.<EBS/EFS/S3> in Metrics Explorer to find all metrics related to that service.

You can view trends by selecting the metric in the metric explorer, specifying the time period, and selecting the aggregation method.

Using SolarWinds Observability, you can also create alerts based on these metrics.

Key Metrics for AWS Storage Options

Let’s briefly examine the key metrics in SolarWinds Observability for the various AWS storage options.

EBS

To understand usage, you should track read and write operations with the following metrics:

EBS.VolumeReadOps: Number of read operations during a specified period.
EBS.VolumeWriteOps: Number of write operations during a specified period.

Regarding cost, EBS pricing is based on the amount of storage and I/O operations per second (IOPS). Look at the total number of bytes transferred for specified operations during a given period:

EBS.VolumeReadBytes: Total number of bytes transferred by read operations during a specified period.
EBS.VolumeWriteBytes: Total number of bytes transferred by write operations during a specified period.

For performance, consider the read/write latency and throughput of EBS volumes:

EBS.AverageReadLatency: Time required to complete read requests during a specified period.
EBS.AverageReadLatency: Time required to complete write requests during a specified period.
EBS.VolumeThroughputPercentage: Percentage of IOPS delivered out of the total IOPS provisioned for a volume.

EFS

Understanding EFS usage and storage patterns requires tracking client connections and read/write operations:

EFS.ClientConnections: Total number of client connections to a file system.
EFS.DataReadIOBytes: Average number of read bytes for each file system read operation.
EFS.DataWriteIOBytes: Average number of write bytes for each file system write operation.

EFS pricing is based on the storage and the amount of data transferred for read/write operations. To track costs, pay attention to these metrics:

EFS.StorageBytes: Average size of the file system in bytes.
EFS.TotalIOBytes: Total number of bytes for each file system operation.

Performance insights come from calculations based on identified anomalies, alerts, and entity status.

sw.metrics.healthscore: Provides insights into EFS's overall health.

S3

To understand S3 usage, track the number of objects stored and requests made to the bucket.

S3.AllRequests: Total number of HTTP requests to the bucket regardless of their type.
S3.NumberOfObjects: Total number of objects stored in a bucket.

S3 pricing is based on the storage and the amount of data transferred during different operations.

S3.BucketSizeBytes: Total amount of data that is stored in a bucket.
S3.BytesDownloaded: Number of bytes downloaded for requests made to a S3 bucket.
S3.BytesUploaded: Number of bytes uploaded for requests made to a S3 bucket.

Understanding S3 performance requires insight into the latency of your S3 buckets.

S3.FirstByteLatency: Time taken to receive the first byte of the response from the bucket for any request made.
S3.TotalRequestLatency: Total time taken from the first byte received to the last byte sent to an S3 bucket.

Conclusion

With the variety of storage options available through AWS, you can make your storage solution decisions based on your specific application use cases and requirements. Your choice of storage will impact the application’s performance, scalability, and cost. Tracking the critical storage metrics is essential to address the application’s performance issues proactively.

The Metrics Explorer from SolarWinds Observability helps monitor the critical metrics from AWS storage services and makes it easy to create alerts. With SolarWinds Observability, you can proactively identify performance bottlenecks, capacity issues, and security vulnerabilities. You can create custom alerts to prevent downtimes, ensuring optimal application performance.