…Cassandra . We're using other open-source infrastructure to help us scale including Hadoop, RabbitMQ, Zookeeper, Thrift, HDFS and Lucene. We're rewriting Digg from the ground up and we need amazing developers to join our world-class team. If you think you are up for the challenge, or you know someone who might be, take a look at our jobs page for more information.
That is when I decided that RabbitMQ might be my best bet, I had heard great things about Erlang's capabilities in the scaling department. After much deliberation and studying, I decided to make the leap and use AMQP and Eventmachine to interact with RabbitMQ. The choice of AMQP was a no brainer, we already used daemon-kit and it has native AMQP support. The switch took less than a weekend to get up and running!
In the end, we went from two starling queue servers …
chef-solr/amqp_password - password for chef vhost in RabbitMQ.
chef-server-webui/admin_password - password for "admin" user in Chef Server WebUI, must be 6 characters.
Preseed settings can be specified with debconf-set-selections .
Reporting Issues
As with other Opscode Open Source projects, please report issues on our JIRA Ticketing system . Create new tickets in the CHEF project with the Packaging component. If you would like to contribute …
excessive memory footprint ( ActiveMQ and RabbitMQ), both at startup (e.g. >120MB for ActiveMQ vs. <3 MB for ocamlmq) and as new queues are created or new topic subscriptions are added
bad performance with ActiveMQ's scalable (to thousands of queues) storage backends ( KahaDB, JDBC)
bad performance in RabbitMQ's topic message dispatch: RabbitMQ was doing a linear scan of the subscription table per dispatch
RabbitMQ did not guarantee that persistent messages …
Available from today, you can now choose either Ubuntu Hardy or Lucid as the base OS when building new Brightboxes.
Ubuntu Lucid is the latest Long Term Support ( LTS) version of Ubuntu with security fixes provided until April 2015. It brings a whole bunch of upgrades such as Ruby 1.8.7, Monit 5, Apache 2.2.14 and new packages like CouchDB, Sphinx, Chef, RabbitMQ, MongoDB, ejabberd and many more.
We've treated it to the usual Brightbox Ruby deployment tune-up, …
Get Your High Scalability Fix at Digg
Interested in working on cutting-edge high-scale infrastructure at Digg? We're making a big investment in scaling and have committed to the NoSQL (Not o nly SQL) path with Cassandra . We're using other open-source infrastructure to help us scale including Hadoop, RabbitMQ, Zookeeper, Thrift, HDFS and Lucene. We're rewriting Digg from the ground up and we need amazing developers to join our world-class team. …
…processing system other than Resque and need to monitor it?
No Scout plugin for your background system? Be like Jesse and learn how to write your own Scout plugin . PS - We've added unit tests to plugins. Make your monitoring scripts rock-solid .
When RabbitMQ is down during the creation of the initial Validation and Web UI keys, we now re-generate them once it is available.
Array's that contain hashes are now indexed properly.
I want to give a quick special thanks to the ton of people who have been helping the entire Chef community get up to speed on the changes in 0.8: by writing documentation, answering questions on the mailing list, and helping people in real-time in IRC . I'm proud to be around each and every …
Given all my work with Fibers and EventMachine over the last three months, it should come as no surprise that I've been working on infrastructure based on Fibers and EventMachine to get maximum scalability without the callback style of code which I dislike for many reasons. Watch my talk on scaling with EventMachine if you need more background on the problem.
Now that I have RabbitMQ, Cassandra, Solr and the Amazon AWS services evented, the only holdup was ActiveRecord…
…needs me to be aware: pick the right CPU architecture, find the AMI with the version of RabbitMQ I'm after etc.
A lot of other ‘cloud' providers (think Joyent Accelerators, Rackspace's CloudServers SliceHost etc.) are very similar. Although you can programmatically control instances, only pay for the time you use them for etc., you're still thinking at a relatively low systems level.
Amazon's Elastic MapReduce Service is an example of a higher-level …