Optimizing AWS Systems Manager Session Manager Logs with Lambda for Better Auditability and Cost Management

AWS Systems Manager Session Manager is a powerful tool for securely managing and interacting with your servers. However, the default session log configuration can generate large log files, especially when commands like tail or less output significant amounts of data. These large files not only increase storage costs but also hinder efficient auditing.

In this blog, we’ll explore a step-by-step solution to optimize these logs using an AWS Lambda function. This function will automatically process session logs, truncate unnecessary output, and store optimized logs in a separate S3 directory. Let’s dive into the implementation!

Understanding the Problem

Session Manager Logs and Their Challenges

When debugging server issues, DevOps engineers often use commands like tail or less to inspect logs. These commands output a large number of lines, which are captured by Session Manager and stored in an S3 bucket. Such extensive logging:

• Inflates storage costs due to excessive file sizes.
• Reduces auditability by cluttering logs with redundant output.

Proposed Solution

We’ll create a Lambda function that automatically processes logs uploaded to S3 by:

• Identifying command outputs (tail, less).
• Removing excessive lines following these commands until a termination pattern is detected (^C, ;username@servername:).
• Replacing the removed content with a summary of lines truncated.
• Saving the processed logs to a separate S3 directory.

Architecture Overview

• Trigger: S3 event notification triggers the Lambda function when a new log is uploaded.
• Processing: The Lambda function processes the log, truncates unnecessary output, and summarizes it.
• Output: The modified log is saved in a separate S3 directory (processed-ssm-logs).

Step-by-Step Implementation

Step 1: Setting Up S3 Bucket

1. Create an S3 Bucket:

• Navigate to the S3 console and create a bucket (e.g., my-ssm-logs-bucket).
• Create two directories within the bucket:
• ssm-logs/: Original logs.
• processed-ssm-logs/: Processed logs.

Step 2: Configuring AWS Systems Manager Session Manager to Deliver Logs to an S3 Bucket

Click Preferences then click Edit in the System Manager Session Manager console:

1. Enable S3 Logging:

• Under the S3 bucket for session logs section:
• Enable the option to log sessions to an S3 bucket.
• Specify the bucket name where logs should be stored (e.g., my-ssm-logs-bucket).
• Optionally, specify a prefix like ssm-logs/ to organize the logs.
• Click Save to apply the changes.

2. IAM Role Permissions for S3 Logging

The Systems Manager Session Manager requires permissions to write to the S3 bucket. These permissions are granted via an IAM role associated with the instance.

a. Locate the Instance Profile Role:

• Identify the IAM role attached to your EC2 instances.

b. Update the Role with S3 Permissions:

• Navigate to the IAM Console.
• Find the relevant role under Roles.
• Attach a policy allowing write access to your S3 bucket.

Example Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-ssm-logs-bucket/ssm-logs/*"
    }
  ]
}

Step 3: Creating the Lambda Function

1. Lambda Function Code

import boto3
import re

s3_client = boto3.client('s3')

def remove_tail_output(content):
    """
    Remove outputs of 'tail' commands from the log content.
    add a message indicating the number of lines removed.
    Stop at ^C, ^c.
    """
    processed_lines = []
    skip_tail_output = False
    tail_line_count = 0
    i = 0

    for line in content.splitlines():
        i += 1
        # Check for start of tail command
        if not skip_tail_output and re.search(r'# (tail|less)', line):
            skip_tail_output = True
            tail_line_count = 0  # Reset the line count for the current tail command
            processed_lines.append(line)  # Keep the 'tail' command line
            print(f"tail/less found at line {i}")
            continue

        # If skipping output, count lines until termination patterns are found
        if skip_tail_output:
            tail_line_count += 1
            if re.search(r'\^c|\^C|;[a-zA-Z0-9._%-]+@[a-zA-Z0-9.-]+:', line):
                skip_tail_output = False
                # Add the termination line and a message about deleted lines
                processed_lines.append(line)
                processed_lines.append(f"*** {tail_line_count} lines of 'tail/less' output removed to save space ***")
                print(f"end found at line {i}, {tail_line_count} lines removed")
            continue

        # Add line if not in skip mode
        processed_lines.append(line)

    return '\n'.join(processed_lines)

def lambda_handler(event, context):
    try:
        # Get bucket and object key from the event
        bucket_name = event['Records'][0]['s3']['bucket']['name']
        object_key = event['Records'][0]['s3']['object']['key']

        # Read the file from S3
        response = s3_client.get_object(Bucket=bucket_name, Key=object_key)
        file_content = response['Body'].read()

        # Attempt decoding with UTF-8, fallback to ISO-8859-1
        try:
            decoded_content = file_content.decode('utf-8')
        except UnicodeDecodeError:
            decoded_content = file_content.decode('latin-1')  # Fallback encoding

        # Process the file content
        processed_content = remove_tail_output(decoded_content)

        # Write the processed content back to the same bucket with a new key
        processed_key = object_key.replace('.dms', '_processed.dms')
        s3_client.put_object(
            Bucket=bucket_name,
            Key=f"processed-ssm-logs/{processed_key}",
            Body=processed_content.encode('utf-8')
        )

        return {
            'statusCode': 200,
            'body': f'Processed file saved as {processed_key}'
        }
    except Exception as e:
        print(f"Error processing file: {e}")
        return {
            'statusCode': 500,
            'body': f"Error processing file: {str(e)}"
        }

Step 4: IAM Role and Permissions

1. Create a Role for Lambda:

• Navigate to the IAM console.
• Create a new role with the AWS Lambda use case.

2. Attach Policies:

1. Inline Policy for Fine-Grained Access:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::my-ssm-logs-bucket/ssm-logs/*",
        "arn:aws:s3:::my-ssm-logs-bucket/processed-ssm-logs/*"
      ]
    }
  ]
}

Step 5: Lambda Function Configuration

1. Basic Settings:

• Allocate 512MB memory(for large log files upto 150MB) and set a timeout of 1 minute (can be adjusted based on log size).

1. Enable Event Notifications:

Click on Add Trigger

Select the required S3 bucket from the drop down.

Step 6: Testing the Setup:

Start a session from the Systems Manager Session manager console, run few commands including tail and less and exit the session.

Check the processed-ssm-logs directory for the processed logs.

Configure to Auto delete original logs files(Optional)

You can configure the lambda function to delete the original log files after the processes file is uploaded to the bucket. This will ensure you only have one copy of the log file.

Conclusion

By implementing this Lambda-based solution, you can significantly optimize your AWS Systems Manager Session Manager logs, reducing storage costs and improving auditability. This approach highlights the power of serverless computing in streamlining DevOps workflows while maintaining compliance and cost efficiency.