People have asked if it is possible to automatically retrieve data stored in an Amazon S3 bucket (such as logs) and load it automatically into Good Data. The answer is a qualified (but loud) Yes! :-)

In order to set this up, Good Data needs full permissions on the files in question. Since Amazon does not allow automatic inheritance of permissions in a specified bucket, we recommend creation of a new S3 account solely for storing the data files. Procedures to set this up are as follows:

1. Create a new S3 account. This account should only be for storing data files.
2. Create a bucket in your new S3 account
   • Please use ONLY lowercase letters and NO underscores ("_") in your S3 bucket name
   • Copy your new S3 account's Access Key and Secret Key, along with the bucket name, into a text file (https://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8&action=access-key)
   • Put this text file into the bucket you just created in this step, and change the ACLs on the file to give FULL_CONTROL to [Good Data account info] (this is the email address of the Good Data S3 account which we will provide upon request)
   • Also give FULL_CONTROL access to this bucket to both the MAIN account email address (your original S3 account) and to [Good Data account info]
3. Once this is successfully set up, you can begin putting your data files into this bucket (in CSV or zipped CSV format), and Good Data will periodically retrieve and load the data into your project.