Related Links

    Add IDrive® e2 S3 storage as Tier Storage in RustFS

    RustFS supports remote tiering, which allows objects in a RustFS bucket to be transitioned to an external S3-compatible storage system based on lifecycle rules. In this configuration, IDrive e2 is used as the remote S3 tier for RustFS.

    With this configuration, RustFS continues to manage the source bucket, while eligible objects are transitioned to the configured IDrive® e2 bucket according to the lifecycle transition rule.

    The configuration consists of three main steps:

    1. Configure the RustFS instance with the rc command-line client.
    2. Configure an IDrive® e2 bucket as a remote RustFS tier.
    3. Create a lifecycle transition rule on the RustFS bucket to transition eligible objects to the IDrive® e2 tier.

    RustFS manages lifecycle rules and remote storage tiers through the rc bucket lifecycle command group.

    Prerequisites

    Before configuring IDrive® e2 as a RustFS remote tier, ensure the following requirements are met:

    • A running RustFS instance with access to the S3 API.
    • The RustFS S3 API endpoint and valid RustFS Access Key and Secret Key.
    • The RustFS rc CLI is installed and accessible from the command line.
    • A valid IDrive® e2 account.
    • An IDrive® e2 bucket dedicated for use as the RustFS remote tier.
    • A valid IDrive® e2 Access Key and Secret Key with the required permissions to access the tier bucket.
    • The IDrive® e2 S3 endpoint and corresponding region for the bucket/access key. IDrive® e2 provides region-specific S3 endpoints and access keys for S3-compatible integrations.
    • Network connectivity from the RustFS server to the IDrive® e2 S3 endpoint.
    • The RustFS bucket on which the lifecycle transition rule will be configured.

    Important Notes

    • The IDrive® e2 tier bucket should be used as a dedicated remote-tier bucket for RustFS.
    • Bucket versioning is not required on the RustFS source bucket or the IDrive® e2 tier bucket for basic object transition.
    • Versioning and noncurrent-version lifecycle rules are only relevant when you want to manage previous object versions.
    • The --transition-days value defines when objects become eligible for transition based on their age.
    • Remote tiering is intended for lifecycle-based storage transition. It should not be considered the same as maintaining an independent backup copy.
    • If you need a separate copy of objects in IDrive® e2 for backup or disaster-recovery purposes, use bucket replication instead of remote tiering.
    • Use HTTPS for the IDrive® e2 endpoint to secure communication between RustFS and IDrive® e2.
    • Keep the RustFS and IDrive® e2 access credentials secure and do not include actual credentials in documentation or scripts.

    1. Configure RustFS S3 Instance

    1. Run the following command with your RustFS credentials to configure the RustFS instance:
      2. # rc alias set rustfs http://localhost:9000 <RustFS Access Key> <RustFS Secret Key>
    2. List the buckets to verify the RustFS connection: 
      # rc bucket list rustfs/

    2. Configure IDrive® e2 as a RustFS Remote Tier

    2.1 Create an IDrive® e2 bucket

    Create a dedicated bucket in IDrive® e2 to be used as the RustFS remote tier.
    For example: 

    rustfs-tier

    2.2 Add IDrive® e2 as a RustFS remote tier

    Run the following command:

                                  # rc bucket lifecycle tier add s3 <Name for Tier> rustfs --endpoint "https://<IDrive e2 Endpoint>" --access-key "<IDrive e2 Access Key>" --secret-key "<IDrive e2 Secret Key>" --bucket "<IDrive e2 Bucket>" --region "<IDrive e2 Region>"

    Example:

    # rc bucket lifecycle tier add s3 E2TIER rustfs --endpoint --endpoint "https://<IDrive e2 Endpoint>" --access-key "<IDrive e2 Access Key>" --secret-key "<IDrive e2 Secret Key>" --bucket "<IDrive e2 Bucket>" --region "<IDrive e2 Region>"

    Expected Output:

    • Tier 'E2TIER' (S3) added successfully.

    2.3 Verify the configured remote tier:

    # rc bucket lifecycle tier list rustfs

    3. Create a Transition Rule Using the IDrive ® e2 Tier

    A transition rule defines when objects in a RustFS bucket become eligible to be transitioned to the configured IDrive® e2 remote tier based on object age.

    3.1 Create the transition rule

    # rc bucket lifecycle rule add rustfs/<RustFS Bucket> --transition-days 30 --storage-class E2TIER

    Example:
    # rc bucket lifecycle rule add rustfs/test-bucket --transition-days 30 --storage-class E2TIER

    Here:
    --transition-days 30 specifies that objects become eligible for transition after 30 days.
    --storage-class E2TIER specifies the IDrive® e2 remote tier created in the previous step.
    <RustFS Bucket> should be replaced with the RustFS bucket to which the transition rule should apply.

    Expected Output:

    • Lifecycle rule '<rule-id>' added successfully.

    3.2 Verify the transition rule

    # rc bucket lifecycle rule list rustfs/<RustFS Bucket>

    Example:
    # rc bucket lifecycle rule list rustfs/test-bucket

    The configured lifecycle rule will transition eligible objects from the RustFS bucket to the IDrive® e2 remote tier after the specified number of days.

    Note:

    • Remote tiering is intended for lifecycle-based storage transition. It should not be considered the same as maintaining an independent backup copy. For maintaining a separate copy of objects in IDrive® e2, use Bucket Replication.
    • Versioning is not required on the RustFS source bucket or the IDrive E2Tier bucket for basic object transition. Versioning is required only if you want to manage noncurrent object versions using lifecycle rules.

    4. Validate Objects Transition

    To validate that the configured lifecycle transition rule is working as expected, verify that the object is available in the IDrive® e2 tier bucket and can still be accessed through RustFS.

    4.1 Verify the Object in IDrive® e2

    After the lifecycle transition is processed, open the rustfs-tier bucket in IDrive® e2 and verify that the transitioned object's data is present in the IDrive® e2 bucket configured as the RustFS remote tier.
    You can also use an S3-compatible client such as rclone to list the contents of the IDrive e2 bucket.

    For example:

    # rclone ls <e2remote>:rustfs-tier

    Verify that the test object is listed in the IDrive® e2 tier bucket.

    Note: The lifecycle transition is processed asynchronously. The object may not appear in the IDrive® e2 bucket immediately after the configured transition period is reached.

    4.2 Verify Object Access Through RustFS

    After the object has transitioned to the IDrive® e2 tier, verify that RustFS can still access the object.

    1. Use the following command to check the object: 
      # rc object stat rustfs/test-bucket/rustfs-tier-test.txt
    2. Download the object through RustFS: 
      # rc object copy rustfs/test-bucket/rustfs-tier-test.txt C:\temp\rustfs-tier-restore.txt
    3. Compare the original and retrieved files: 
      # Get-FileHash C:\temp/rustfs-tier-test.txt
      # Get-FileHash C:\temp/rustfs-tier-restore.txt

    The hashes should match, confirming that the object can be successfully retrieved through RustFS after the lifecycle transition and that the object's content remains intact.

    5. Update Tier Credentials

    To rotate the IDrive® e2 Access Key and Secret Key used by the RustFS remote tier, create the replacement credentials first and update the configured tier before revoking the old credentials.

    5.1 Create Replacement IDrive® e2 Credentials

    Create a new IDrive® e2 Access Key and Secret Key with the required permissions for the configured tier bucket.

    Do not revoke the existing credentials at this stage. Keep them active until the new credentials have been successfully validated.

    5.2 Update the Configured Tier

    Update the E2TIER configuration with the new IDrive® e2 credentials:

                                     rc bucket lifecycle tier edit E2TIER rustfs `
    
                                 --access-key "<NEW_IDRIVE_E2_ACCESS_KEY>" `
    
                                 --secret-key "<NEW_IDRIVE_E2_SECRET_KEY>"

    5.3 Verify the Updated Tier

    Verify that the tier configuration has been updated successfully:

    rc bucket lifecycle tier info E2TIER rustfs

    Confirm that the tier is configured with the expected IDrive® e2 settings.

    You should also validate access to a transitioned object to confirm that RustFS can communicate with the IDrive® e2 tier using the new credentials.

    5.4 Revoke the Old Credentials

    After the new credentials have been successfully validated, revoke the old IDrive® e2 Access Key.

    Important: Do not revoke the old credentials before validating the new credentials. Revoking them prematurely may make previously transitioned objects unavailable if RustFS cannot access the configured IDrive e2 tier.

    6. Removing Transition Rule (Life cycle rule)

    Before removing a lifecycle transition rule, verify that no objects still depend on the rule for future transitions and that any objects already transitioned to the IDrive® e2 tier remain accessible.

    6.1 Identify the Lifecycle Rule

    Run the following command

    # rc bucket lifecycle rule list rustfs/<RUSTFS_BUCKET>

    ID Status Prefix Expiry Transition Storage Class
    rule-8c07e488 Enabled - - 30 day(s) E2TIER
    rule-096fdeb0 Enabled - - 1 day(s) E2TIER

    Identify the Rule ID of the lifecycle rule that you want to remove.

    6.2 Disable the Lifecycle Rule

    Disable the rule before removing it: 

    # rc bucket lifecycle rule edit rustfs/<RUSTFS_BUCKET> --id <Rule ID> --disable true

    Expected Output:

    • Lifecycle rule updated successfully.

    6.3 Remove the Lifecycle Rule

    After confirming that the rule is no longer required, remove it:

    # rc bucket lifecycle rule remove rustfs/<RUSTFS_BUCKET> --id <RULE_ID>

    Expected Output:

    • Lifecycle rule 'rule-8c07e488' removed.

    Note: Removing or disabling a lifecycle transition rule does not automatically restore objects that have already been transitioned to the IDrive E2TIER.

    7. Removing Remote Tier

    Before removing a remote tier, ensure that no lifecycle rules or transitioned objects still depend on the configured IDrive E2TIER.

    7.1 Complete the Required Checks

    Before removing E2TIER: (Explained with the name of the configured remote tier for testing)

    1. Disable all lifecycle rules that reference E2TIER.
    2. Verify that no pending lifecycle transition activity is in progress.
    3. Identify objects that have already been transitioned to the IDrive.
    4. Restore or migrate dependent objects as required.
    5. Verify that no transitioned objects still depend on the IDrive E2TIER for access.
    6. Back up the relevant tier and lifecycle configuration for reference or recovery purposes.
    7. Keep the IDrive E2TIER bucket available until all dependent objects have been verified as recoverable.

    7.2 Remove the Remote Tier

    After completing the above checks, remove the configured remote tier: 

    # rc bucket lifecycle tier remove E2TIER rustfs

    Verify that the tier has been removed: 

    # rc bucket lifecycle tier list rustfs
    The e2 TIER entry should no longer appear in the list of configured remote tiers.

    Operational Restrictions

    When using IDrive® e2 as a remote tier for RustFS, follow these operational restrictions to avoid data-access or lifecycle issues:

    1. Do not rename, overwrite, or delete tier objects directly in IDrive® e2. Tiered objects should be managed through RustFS.
    2. Do not use the IDrive® e2 tier bucket for unrelated application data. Keep the bucket dedicated to RustFS tier storage.
    3. Do not remove the configured tier while transitioned objects depend on it. Ensure dependent objects are restored or otherwise made independently recoverable before removing the tier.
    4. Do not change the configured endpoint, bucket, or region without a planned migration procedure. Changes may prevent RustFS from accessing previously transitioned objects.
    5. Do not use forced tier removal unless all transitioned objects are independently recoverable.
    6. Protect the IDrive® e2 bucket and credentials. Loss of access to the configured tier bucket or its credentials can make transitioned objects unavailable through RustFS.