AWS S3 & NodeJS to Change Storage Class

Backing Up to AWS S3

If like me you want to store your old computer files, photos or videos in the cloud and you are using AWS S3 then you might have uploaded them using the default STANDARD storage class. This is fine if you just have infrequently accessed files, but the trouble is it can end up costing you way more than you need.

Also, maybe you already have some files in an S3 Bucket using the lowest cost DEEP_ARCHIVE tier but you have uploaded some more files using the STANDARD tier and forgot to change the class.

AWS S3 can be amazing, but if you do not stay on top of managing your buckets then you will easily forget to change the storage class of your files, and a few months down the line wonder why your costs have shot up!

Scanning the S3 Bucket

To find out what storage class your files are using you can use the AWS CLI to scan the bucket.

aws s3api list-objects  \
    --profile YOUR_PROFILE  \
    --bucket YOUR_BUCKET  \
    --query 'Contents[].{Key: Key, StorageClass: StorageClass}'

This returns a JSON array of all the files in the bucket and their storage class... which is fine for looking at each file.

Now lets instead use a NodeJS Script that Counts the Files in each Storage Class making it much easier to give us a better breakdown of how many files for each storage class.

const PROFILE = 'YOUR_PROFILE';
const BUCKET = 'YOUR_BUCKET';

const AWS = require('aws-sdk');
const credentials = new AWS.SharedIniFileCredentials({ profile: PROFILE });// Load the AWS profile from ~/.aws/credentials
const s3 = new AWS.S3({apiVersion: '2006-03-01',credentials,});


const counts = {};

function handleListObjectsResponse(response) {
  response.Contents.forEach(function(obj) {
    const storageClass = obj.StorageClass;
    if (!counts[storageClass]) {
      counts[storageClass] = 0;
    }
    counts[storageClass]++;
  });
  if (response.IsTruncated) {
    s3.listObjectsV2({
      Bucket: BUCKET,
      ContinuationToken: response.NextContinuationToken
    }, function(err, data) {
      if (err) {
        console.log('Error:', err);
      } else {
        handleListObjectsResponse(data);
      }
    });
  } else {
    console.log(counts);
  }
}


s3.listObjectsV2({ Bucket: BUCKET }, function(err, data) {
  if (err) {
    console.log('Error:', err);
  } else {
    handleListObjectsResponse(data);
  }
});

This will give you an output a bit like this:

{ 
    DEEP_ARCHIVE: 387720, 
    GLACIER: 84255, 
    STANDARD: 24117 
}

So I know that I have a load of STANDARD files that I need to move to DEEP_ARCHIVE...

Changing the Storage Class

This next script goes through our bucket and changes the storage class of any files that are STANDARD to DEEP_ARCHIVE.

const PROFILE = 'YOUR_PROFILE';
const BUCKET = 'YOUR_BUCKET';
const storageClassFrom = 'STANDARD';
const storageClassTo = 'DEEP_ARCHIVE';

const AWS = require('aws-sdk');
const credentials = new AWS.SharedIniFileCredentials({ profile: PROFILE });
const s3 = new AWS.S3({apiVersion: '2006-03-01',credentials,});

function handleListObjectsResponse(response) {

  response.Contents.forEach(function(obj) {
    const storageClass = obj.StorageClass;
    if (storageClass === storageClassFrom) {
      s3.copyObject({
        Bucket: BUCKET,
        CopySource: `${BUCKET}/${obj.Key}`,
        Key: obj.Key,
        StorageClass: storageClassTo,
      }, function(err, data) {
        if (err) {
          console.log('Error:', err);
        } else {
          console.log(`Changed storage class of object ${obj.Key} from ${storageClassFrom} to ${storageClassTo}`);
        }
      });
    }
  });
  // If there are more objects, make another listObjectsV2 call to get the next page
  if (response.IsTruncated) {
    s3.listObjectsV2({
      Bucket: BUCKET,
      ContinuationToken: response.NextContinuationToken
    }, function(err, data) {
      if (err) {
        console.log('Error:', err);
      } else {
        handleListObjectsResponse(data);
      }
    });
  }
}

// Get a list of all objects in the bucket
s3.listObjectsV2({ Bucket: BUCKET }, function(err, data) {
  if (err) {
    console.log('Error:', err);
  } else {
    handleListObjectsResponse(data);
  }
});

Please note - if your bucket contains many files this might take you a while to run... so be patient! It took about 30 mins for each bucket I ran it on, as they had quite a few files!

I recently ran this script on two of my buckets that I knew still had a lot of STANDARD files in them as I had uploaded a load of files over a month ago but had forgotten to set them to DEEP_ARCHIVE. It managed to save me a daily cost saving of around $0.37 per day.

Conclusion

I hope this post has helped you to save some money on your AWS S3 bills. If you have any questions or comments please contact me on Twitter at https://twitter.com/joemore_g