e3 Bucket Lifecycle Rules (Automatic Object Expiration)

In this guide we explain how to use bucket lifecycle rules with the e3 object storage service so you can automatically manage the object lifecycle. We'll create an example rule to expire (delete) objects that are older than a specified number of days.

What are Bucket Lifecycle Rules?

Lifecycle rules allow you to automate tasks for objects stored in your e3 buckets. By defining lifecycle rules, you can tell the e3 service to automatically perform actions, such as expiring old objects, based on their age or other criteria. This can help you save on storage costs and keep your data organized without manual intervention.

Key Components of an Object Expiration Rule

When creating a rule to expire objects, you'll define a few key pieces of information:

• ID: A unique name to identify your rule (e.g., "ExpireOldLogFiles").

• Status: Set to "Enabled" to activate the rule, or "Disabled" to keep it inactive.

• Filter: Specifies which objects the rule applies to.

  • Prefix: You can apply your rule to all objects in the bucket by setting Prefix to an empty string (""), or you can target objects in a specific "folder" or with a certain name pattern (e.g., "logs/", "archive/").

• Expiration Action: This is the part that tells the e3 service to delete objects.

  • Days: You specify the number of days after an object's creation date when it should be considered expired and marked for deletion.

How e3 Processes Lifecycle Rules

The eazyBackup e3 object storage service has an automated process that periodically checks for objects that meet the criteria of your rules. When an object is eligible for expiration (e.g., it's older than the specified number of days), the service will schedule it for deletion. This process typically runs daily.

Creating a Lifecycle Rule: Expire Objects After X Days

Lifecycle configurations are applied to a bucket using S3-compatible tools, such as the AWS CLI. These tools usually expect the configuration to be provided in a JSON file. At the time of writing, lifecycle policies cannot be configued from our e3 Storage Dashboard.

Let's create a rule to expire all objects in your-bucket-name after X_DAYS (e.g., 30, 90, 365 days).

1. Replace Placeholders:

   • X_DAYS with the desired number of days for objects to live before expiration.

   • your-rule-id with a descriptive ID for your rule (e.g., "ExpireAfter90Days").

2. Create a JSON Configuration File: Create a file (e.g., lifecycle-expire-config.json) with the following content:

{
  "Rules": [
    {
      "ID": "your-rule-id",
      "Filter": {
        "Prefix": ""  
      },
      "Status": "Enabled",
      "Expiration": {
        "Days": X_DAYS 
      }
    }
  ]
}

Example for 90 days:

{
  "Rules": [
    {
      "ID": "ExpireObjectsAfter90Days",
      "Filter": {
        "Prefix": ""
      },
      "Status": "Enabled",
      "Expiration": {
        "Days": 90
      }
    }
  ]
}

Applying the Lifecycle Configuration

You can apply this configuration to your bucket using an S3-compatible tool like the AWS CLI, configured with your eazyBackup e3 service endpoint and credentials.

1. Save your JSON configuration lifecycle.json.

2. Use a command similar to the following, replacing placeholders:

aws s3api put-bucket-lifecycle-configuration \
    --bucket your-bucket-name \
    --lifecycle-configuration file://lifecycle.json \
    --endpoint-url <e3_endpoint_url> \
    --profile <e3_profile>

• Replace your-bucket-name with the actual name of your bucket.

• Replace lifecycle.json with the path to your JSON file.

• Replace <e3_endpoint_url> with the service endpoint for eazyBackup e3 found on the Access Keys page of the Cloud Storage Dashboard.

• Replace <e3_profile> with your AWS CLI profile name if you use one.

Important Considerations:

• From Creation Date: The Days for expiration are counted from the time an object was originally created (uploaded) in the bucket.

• Periodic Processing: Object expiration is not instantaneous once an object reaches its expiry age. The e3 service processes lifecycle rules periodically (usually daily). An object might remain in the bucket for a short period after its official expiry time until the next processing cycle.

• Versioning: If versioning is enabled on your bucket, the Expiration action for current object versions typically creates a "delete marker." You would then need separate rules (e.g., NoncurrentVersionExpiration) to manage the deletion of old, non-current versions and the delete markers themselves if you want to reclaim storage from them. If versioning is not enabled, expired objects are permanently deleted.

• Testing: Test your lifecycle rules with non-critical data or on a test bucket first to ensure they respond as expected.

• Modifying Rules: To change a lifecycle rule, you would typically modify your JSON configuration file and just re-apply it using the same put-bucket-lifecycle-configuration command. This overwrites the existing lifecycle configuration on the bucket. To remove lifecycle rules, you can use the delete-bucket-lifecycle command.

Using lifecycle rules will help you manage your data within the e3 storage service.