I discuss S3 here but other cloud based storage can be used as well. S3 is just the most popular in this category and is in wide use.
Details:
s3cmd - we have been using the version from github , Mostly for multi-part upload support. This prevents us from having to split files up before uploading to S3.
There is released alpha version of this version here
You can now set bucket lifecycle properties so data over X days is archived to Glacier and data over Y …
- JJinuxLand: Trip to "reflections projections|2012"
- Engine Yard Developer Blog: Orchestra Elastic Apps For Everyone
- Luigi Montanez: Developer Happiness and MongoDB
- High Scalability: Stuff The Internet Says On Scalability For February 29, 2013
- Tammer Saleh: Bushido
- 41 technologies: Building a Ruby on Rails application in a week
…quickly deploy and manage apps within AWS's cloud of services ( EC2, S3, etc.) It now supports Ruby apps by using Phusion's Passenger (I hope Amazon are paying Phusion handsomely for this :-))
Reading
Yet Another Ruby Shootout: MRI 1.9.3 vs MRI 2.0 vs Rubinius 2.0
Igor Alexandrov picks up where Antonio Cangiano left off by benchmarking MRI 1.9.3-p286, MRI 2.0.0-preview1 and Rubinius 2.0.0-rc1. As always with benchmarks, don't read too …
- Plataformatec Blog: Eager loading for greater good
- The Pug Automatic: SendGrid metadata and Rails
- A Fresh Cup: What's New in Edge Rails #27
- Viget.com Blogs: A Better Way to Add a ‘selected' Class to Links in Rails
- giant robots smashing into other giant robots: Writing Better Cucumber Scenarios; or, Why We're Deprecating FactoryGirl's Cucumber Steps
- New Relic: Rails Machine Adds APM from New Relic to its App Management Platform for Rails
…happens to costs when the system grows? e.g. Pinterest has around 410TB of data on S3, what if that keeps growing at a rate of 25% every month, like it has been in the last 10 months?
I created a couple of deployments in PlanForCloud to explore these questions and the results are very interesting - they show how important it is to get an idea of costs and budget for cloud spend prior to using the cloud :
Their main data is stored in S3.
All their web stuff is in Python. They use Pylons, but they have evolved very far away from Pylons. They also have their own templating system.
MySQL really doesn't like supporting more than 100 open connections.
When they receive a DMCA takedown, they don't try to delete it from the user's account entirely. They just ban the item from being publicly shared.
A lot of people at Dropbox came from Google.
My talk: Structured Web Programming…
- Zach Inglis: The Jocks & The Geeks
- the evolving ultrasaurus: nreduce: self help for startups?
- » Product Launch Blog: Moving Your Apps to Mobile: Part 3 - Mobile Opportunities
- Rails Rumble: 2010: The Year We Made Rumble History!
- Kalzumeus Software: Hacking Customers' Technology Adoption Cycles
- dealloc.me -- Justin Palmer: Programming Politics
…understand is why you aren't more upfront with your customers about it. After all, the last time S3 went down, your status board showed all green except for a tiny icon that indicated something might actually be amiss . Just a reminder:
After the guff you took from the last outage, you'd think that you'd be a little bit more upfront with customers (i.e. the internet). But there were no Twitter updates (the last one tweet was two days beforehand ). At first …
- High Scalability: Stuff The Internet Says On Scalability For June 15, 2012
- Kalzumeus Software: Kalzumeus Podcast 3: Growing Consulting Practices, with Brennan Dunn
- Lessons Learned: A new field guide for entrepreneurs of all stripes
- Ruby Inside: Amazon Releases aws-sdk, An Official AWS SDK for Ruby Developers
- OnRails.org: Amazon RDS: Amazon Relational Database Service or MySQL in the Cloud for Ruby On Rails.
- Engine Yard Developer Blog: The Art of Library
…got a great status dashboard, showing history and uptime. We use it all the time and depend on its S3 service for our products. Sure, there have been some shaky launches in the past (looking at you , EC2). Overall, however, we haven't had many problems — except until recently. It was a good example of how transparency, or the lack of transparency rather, can damage customer trust.
S3 went down last week. So we checked the status dashboard to see what was up. Everything was reported …
- ZURB: Lay Your Major Feature Changes On The Table
- ZURB: Hey Amazon, Gotta Be More Upfront With Your Customers
- High Scalability: Stuff The Internet Says On Scalability For July 1, 2011
- Trevor Turk: Links for 09-26-11
- JJinuxLand: Amazon Web Services Summit 2011: Navigate the Cloud
- Freelancing Gods: A Sustainable Flying Sphinx?
…working with a bunch of large files, though, you're probably better served by uploading them to S3 or RackSpace Cloud Files and serving them from there.
Another option to look at might be Dragonfly , which takes a different approach to photo processing than does Paperclip, resizing on the fly rather than on upload. This might obviate the need for Resque but at unknown (by me) cost. We hope that some of this will be helpful in your next photo-intensive project.
- upstream agile - software: Http Caching assets in Rails 3.1
- giant robots smashing into other giant robots: Backbone.js on Rails hits 1.0
- Lessons Learned: Doing > Talking
- I can't code! Now what?: Hype: the silent evil that looms.
- » Product Launch Blog: Does it pay to invest in firms that design their products well?
- Signal vs. Noise: Laver's Law of Fashion
Missing out on a day in the woods. C'est la vie.
Configster - Configuration management utility that stores things in an external YAML file for easy end-user editing.
S3itch - If you don't like Skitch's switch to Evernote storage, you can use this to move your files to S3 instead.
PLine - Line-oriented profiler for Ruby 1.9.2/1.9.3.
A guide to the basics of jQuery - Online fundamentals course with examples you can play with.
What follows is the story of a bug we encountered during development of what will become the Chef 11 API server. The story unfolds as we began to integrate and test Bookshelf, a new component that handles cookbook file storage. Bookshelf exposes a REST API that behaves just enough like the AWS S3 API to allow the other parts of Chef that make use of it to seamlessly switch between Bookshelf and S3.
We began integrating Bookshelf into the Chef Server once we had passing integration …
…Elastic Block Storage ( EBS) volumes for application and database storage and we use S3 for backups. Additionally we take snapshots of each of our customers' EBS volumes at configurable time intervals. We currently do not utilize ephemeral storage for any platform features.
The hi1.4xlarge instance type is making a splash in the cloud computing community as it is the first to offer solid-state drives ( SSDs) instead of the more traditional spinning disk drives for ephemeral storage. …
- New Relic: Call for experts: Are you a developer or engineer who wants to share best practices? Write for this blog!
- Concentration de studioConcentration de studio: fungazi: Ellie Goulding - High For This (Weeknd...
- A Fresh Cup: What are All Those Rails Gems?
- Forum Discussions - Enterprise Rails: A lot of postgresql database users authenticating and connecting from an enterprise rails application
- Lessons Learned: Learning is better than optimization (the local maximum problem)
- Global Nerdy: A "Whiteboard Jam" on Monads and Coordinate Systems