…sidekiq worker servers, and every one has 4 processes which every one has 25 workers. As we are all on EC2, we use spot instances for part of the cluster so we can add more workers as needed, so it could grow up from 400 to 800 or more.
The technology is nothing special but solid, Rails + Redis + Sharded MySQL.
Thanks Libin and everyone else who entered. I hope Sidekiq Pro makes your system even better!
- Ruby Inside: The Last 2 Weeks in Ruby: Rails 3.2.6, Savon 1.0, RailsInstaller for OS X, Pow 0.4.0, Rails 2 Asset Pipeline
- New Relic: Thread Safe APIs and Sidekiq Support for Your Threading
- RailsCasts: #366 Sidekiq
- A Fresh Cup: Double Shot #999
…physical resources to a hybrid infrastructure comprised of physical servers and Amazon EC2 cloud resources, it encountered a number of challenges:
Create application development pipeline integrating physical servers, Amazon EC2 resources, Capistrano for deploying application updates, and Vagrant for managing environments across different phases of development, testing, and production
Establish transparency throughout entire development workflow and across development team to reduce …
- MySQL Performance Blog: Clarification on MySQL security vulnerability
- Engine Yard Developer Blog: High I/O on Engine Yard Cloud
- High Scalability: Strategy: Use Spare Region Capacity to Survive Availability Zone Failures
- New Relic: Amazon + New Relic FTW!
- UK Specialists in Ruby on Rails, Exalead, AWS, Consultancy - Blog: Amazon wishlist - Please give us an EC2 Recovery Console!
- A Fresh Cup: Double Shot #899
…backups, and if you have a test environment, load them there periodically. You can also spin up an EC2 instance to load your backups onto. In addition, you can binlog rollforward 24 hours of binlogs as a good test.
Store your binlogs off your primary server so you can perform point in time recovery.
Store your binlogs offsite for disaster recovery scenarios.
Run pt-table-checksum periodically (i.e. once a month) and make sure your servers data stays consistent. Checksumming is important, …
- Luigi Montanez: Developer Happiness and MongoDB
- JJinuxLand: Trip to "reflections projections|2012"
- Engine Yard Developer Blog: Orchestra Elastic Apps For Everyone
- High Scalability: Stuff The Internet Says On Scalability For February 29, 2013
- Tammer Saleh: Bushido
- Trevor Turk: Automated Heroku Backups
…Hosted Chef™ , automating the configuration of their entire EC2 environment.
Managing Meteor's sophisticated, primarily Windows-based infrastructure presented a number of challenges as the launch date for HAWKEN approached:
- Create mirrored staging and production environments using EC2 resources
- Ensure 100 percent uptime for HAWKEN game platform
- Adapt to changes in cloud resource availability on the fly, all with only …
- Engine Yard Developer Blog: Using Chef for DevOps Innovations
- blog.webandy.com | Andy Atkinson: Twin Cities Code Camp 2006
- High Scalability: Analyzing billions of credit card transactions and serving low-latency insights in the cloud
- DigitalHobbit: Rails in the Cloud: AWS, Heroku, and Morph
- Ruby Inside: Amazon Releases aws-sdk, An Official AWS SDK for Ruby Developers
- giant robots smashing into other giant robots: Giving Thanks
At the time I left Songkick , it took 1.5 hours to run all the cukes and rspec ‘unit' tests on the big ball of Rails . We were already parallelising over a few in-house VMs at the time to make this manageable, but it still took 20 minutes or so to get feedback. After I left, the team worked around this by getting more slave nodes from EC2, and the build time went down to under 10 minutes.
Then guess what happened?
They added more features to the product, more tests …
- A Fresh Cup: Double Shot #1079
- Trevor Turk: Weekly Digest, 12-11-09
- Spyced: Why your data may not belong in the cloud
- the evolving ultrasaurus: freedom in ruby and code
- Ruby Inside: 18 New Ruby and Rails Jobs for April 2011
- Jake Scruggs: Rails Conf 2010 Day 2
…customer support portal at admin.example.com. Maybe it is a publicly accessible staging server at EC2 spun up by a developer who has since left the company and not shut down because, hey, $ 20 a month. Maybe it is a 20% project by a junior engineer which he has on the back burner for the moment. It doesn't matter why this app was forgotten: if it is publicly accessible, it will be compromised.
What was the proper way to react to these patches? Patch immediately, install a workaround …
- Matthew Bass: Excluding iPads from the mobile-fu Rails plugin
- Hobo Blog: Learn Hobo and Write Documentation
- A Rubyist Railstastic Adventure: Lies That Ruby on Rails People Tell You
- Giles Bowkett: BritRuby Kerfuffle: Silly Even By Ruby Standards
- Blog - James Mead: Using Mocha with Rails 3 and Test::Unit
- Binary Code: Bootstrap, Rails and Small Input Fields
…is simply under a great deal of traffic and you're getting CPU bound. On Amazon EC2, you could even be experiencing slow EBS I/O (since EBS is a network-based storage system). Regardless, when you see long periods of sustained request queuing, it's often useful to rule out that *any* external resource on which your application depends is the bottleneck. Generally speaking, you'll usually find that something is indeed slowing things down if you look hard enough.
If you see large …
- RailsLab: State of the Stack: A Ruby on Rails Benchmarking Report - 05 October 2010
- Brightbox Ruby Blog: New Relic Agent vulnerability
- Lindsay Holmwood - auxesis' musings: Instrumenting your monitoring checks with New Relic
- Ruby Inside: New Relic Releases Real User Monitoring and Ditches RPM Name
- Opscode Blog: Opscamp San Francisco
- Heroku: New Relic RPM Silver & Gold Add-on
But, with EC2 there isn't yet an equivalent - if you've used an EBS backed instance then you can stop your instance, mount the volume elsewhere and attempt to diagnose and fix any issues - there is a guide over here . This doesn't allow you to drop into single user mode, and is a bit more effort - it would be great if Amazon could provide something more like LISH...
Why would it be useful? Sometimes you can be a dork and make changes to the system that don't …
- Medium eXposure: My Tools of the Trade
- Engine Yard Developer Blog: Welcome to Cloud DNS
- Zerosum Dirt(nap): Can You Feel The Rumble?
- Smarticus: Moved the blog once again
- Rails Rumble: Just When You Thought You Were Out...
- giant robots smashing into other giant robots: Hoptoad is on AppSumo's November Developer Bundle
We are doing a migration from Amazon RDS to EC2 with a customer. This, unfortunately, involves some downtime - if you are an RDS user, you probably know you can't replicate an RDS instance to an external server (or even EC2). While this is annoying, this post isn't going to be a rant on how RDS can make you feel locked in. Instead, I wanted to give you a quick tip.
So here's the thing - you can't stop replication on RDS read replica: replica> STOP SLAVE; ERROR 1045 …
- Topics: Amazon.com
- Coding Horror: Building Servers for Fun and Prof... OK, Maybe Just for Fun
- High Scalability: Strategy: Guaranteed Availability Requires Reserving Instances in Specific Zones
- Engine Yard Developer Blog: Let the Engine Yard Cloud dashboard and SSH access help you
- Ruby Inside: The Past 2 Weeks in Ruby: 1.9.3-p286, JRuby 1.7RC2, Sidekiq Pro and Much More
- paperplanes: Web Operations 101 For Developers
- Paul Dix Explains Nothing: Using the Nginx Memcached module with Passenger
…stack from AWS. In fact, it's bringing together the traditional components from EC2, S3, RDS and CloudWatch into a fully scalable infrastructure. So if your application is running on AWS already, it's very likely you can take advantage of Elastic Beanstalk's orchestration and scaling system right now. Elastic Beanstalk supports applications running on Java, Python, PHP and Ruby running on Linux, or IIS running on Windows Server. It also supports databases …
- High Scalability: PaaS on OpenStack - Run Applications on Any Cloud, Any Time Using Any Thing
- Engine Yard Developer Blog: Simple DNS for AppCloud