Sriram Krishnan was enquiring about distributed programming tools and all things distributed. No doubt, the first thing that came into my mind was "erlang", and taking into consideration that he was inviting feedback & his recent work on cacheman and on server /app performace i thought it would be a good time to talk about bspawner - my own pet project among other things erly.
Load Balancing & Introducing bspawner
Erlang nodes can communicate over the network if they find each other. You can send a message to any process (differ from OS process) through its Pid – even if the process is on another node.
Although you don't have to know where processes are, it is still up to the erlang programmer to decide which node initially spawns a task on which node as there is no inherent way to make use of multiple nodes , even though they recognize each other & can message-pass between them.
bspawner is a project i've open sourced - that attempts to load-balance the task of spawning across multiple nodes . The steps involved can be isolated into a couple of distinct problems.
- deciding which node needs to spawn a task
- communicating across these nodes
- maintaining a record of nodes, added /removed nodes ,etc
This project in its essence, deals with the first part and the implementation of the message passing begins with the "messenger.erl" sample program and modifiedto handle the intended message-passing, load-balancing and node-information
Cheers to the feedback, suggestions , comments, and further changes inspired by the growing involvement of the erlang community & encouragement from #erlang in particular.
However, since the project is still in its early stages - and although it was a fantastic learning experience - I learned that Erlang already had inbuilt load-balancing module that I quote:
How cool can it get! Since it's even build in a master-slave basis - when one worker goes down, restart strategies can be configured courtesy of Erlang's OTP supervisor and behaviours. this basically allows you to horizontally scale and have distribute processing among box's. I'm even wondering to test this setup at hover.in on weekends by maybe doing some bizarrely wild clustered processing job like ...pool can be used to run a set of Erlang nodes as a pool of computational processors. It is organized as a master and a set of slave nodes and includes the following features:
- The slave nodes send regular reports to the master about their current load.
- Queries can be sent to the master to determine which node will have the least load.
- finding the largest prime number in Pune maybe ? :D
- any thing from distributing cron jobs to non-blocking or blocking I/O to ... free your mind!
Testing & Test suites
I'm not much into the whole test suite setup as yet, but there a good test suite that is widely adopted would be
- EUnit - a Lightweight Unit Testing Framework for Erlang
- An excellent error report evaluation, testing and debugging paper can also be found here (although a little old '92 , still gives valuable info on general practices)
Distributed Monitoring & Debugging
Regarding utilities for debugging & profiling ,etc apart from user-contributed packages, and there are loads of them on process-one, trapexit , jungerl, google code ,etc - I thought I would list few of the interesting utilties . ( see more on the left side of the documentation at erlang.org under Tool Applications )
- appmon — a graphical utility to observe and manipulate supervisiontrees.
- debugger — an Erlang source code debugger.
- erl_interface — a set of libraries for communicating with distributed Erlang nodes.
- et — the event tracer and tools to record and give a graphical presentation of event data.
- eva — the “event and alarm” handling application.
- observer — tools for tracing and observing the behaviour of a distributed system.
- os_mon — a tool to monitor resource usage in the external operating system.
- pman — a graphic tool to inspect the state of the system, at local or remote Erlang nodes.
- runtime_tools — miscellaneous small routines needed in the runtime system.
- toolbar — a graphical toolbar from which applications can be started.
- webtool — a system for managing web-based tools (such as inets)
- tools — a package of stand-alone applications for analysing and monitoring Erlang programs. This includes tools for profiling, coverage analysis, cross reference analysis etc.
Preserving State in Datastrutctures, Process's or Servers
Behaviours are templates or formalizations of common design patterns. The three inbuilt behaviours are gen_server (client-server paradigms), gen_event (event-driven paradigms) and gen_fsm ( finite state machine paradigms). In addition you can create your own behaviours and have modules implement them. (Emacs in erlang mode, even gives nice skeletons for all behaviours, common design patterns)
You can also have stateless or stateful process's which can run indefintely, or be blocking in nature or exit after a timeout. Which really opens up a whole lot of possibilities, but again - since the key is in message passing - regardless of the process's being on your local node or half way across the globe.
To think about ....
- load- balancing two or multiple yaws servers each of which are capable of handling 80,000 parallel connections each.
- Having erlang communicate via a port to say a python or perl for doing abstracting cross-language functionaliry ( Facebook Chat seems to be the largest xmpp-erlang based web application , and communicates with c++ for logging)
Keep Clicking,
Bhasker V Kode
