SPREAD: A Reliable Multicast and Group Communication Toolkit
-----------------------------------------------------------

/===========================================================================\
| The Spread Group Communication Toolkit.                                   |
| Copyright (c) 1993-2004 Spread Concepts LLC                               |
| All rights reserved.                                                      |
|                                                                           |
| The Spread package is licensed under the Spread Open-Source License.      |
| You may only use this software in compliance with the License.            |
| A copy of the license can be found at http://www.spread.org/license       |
|                                                                           |
| This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF     |
| ANY KIND, either express or implied.                                      |
|                                                                           |
| Spread is developed at Spread Concepts LLC and the Center for Networking  |
| and Distributed Systems, The Johns Hopkins University.                    |
|                                                                           |
| Creators:                                                                 |
|    Yair Amir             yairamir@cs.jhu.edu                              |
|    Michal Miskin-Amir    michal@spreadconcepts.com                        |
|    Jonathan Stanton      jstanton@gwu.edu                                 |
|                                                                           |
| Major Contributors:                                                       |
|    Cristina Nita-Rotaru crisn@cs.purdue.edu - GC security.                |
|    Theo Schlossnagle    jesus@omniti.com - Perl, skiplists, autoconf.     |
|    Dan Schoenblum       dansch@cnds.jhu.edu - Java interface.             |
|    John Schultz         jschultz@d-fusion.net - contribution to process   |
|					          group membership.         |
|                                                                           |
| Contributors:                                                             |
|    Ryan Caudy        wyvern@cnds.jhu.edu - Group membership               |
|    Ben Laurie	       ben@algroup.co.uk - FreeBSD port and warning fixes   |
|    Daniel Rall       dlr@finemaltcoding.com - Java & networking fixes,    |
|                                               configuration improvements  |
|    Marc Zyngier                        - Windows fixes                    |
|                                                                           |
| Special thanks to the following for providing ideas and/or code:          |
|    Ken Birman, Danny Dolev, Mike Goodrich, Ben Laurie,                    |
|    David Shaw, Robbert VanRenesse.                                        |
|                                                                           |
| Partial funding provided by the Defense Advanced Research Projects Agency |
| (DARPA) and The National Security Agency (NSA) since 2000. The Spread     |
| toolkit is not necessarily endorsed by DARPA or the NSA.                  |
|                                                                           |
| WWW    : http://www.spread.org  and  http://www.cnds.jhu.edu              |
| Contact: spread@spread.org                                                |
|                                                                           |
| Version 3.17.3 built 15/October/2004                                      |
\===========================================================================/

October 15, 2004 Ver 3.17.3
---------------------------

1) Fix memory leak in Skiplist. Reported by Taj Khattra, patch by Theo 
   Schlossnagle.
2) spuser,spflooder, and spmonitor fixed to print correct name in help. 
   Patch by Daniel Rall.
3) Fix incorrect alarm printing where WARNING messages generated by
   older Alarm() interface were not printed.
4) Modify monitor so it will run correctly on Windows. This was done by
   modifying monitor.c so it can also build as a threaded program and
   on windows will use multiple threads to send/recv updates and get
   user imput. (A POSIX thread option is also added).
5) Fix a bug in the Windows build using VC++ of thread-safe libtspread.lib.
   Bug and fix by Jacob Green.
6) Fix bug in libspread where if the groups array or message body passed
   to SP_recv* was too small, the mess_type field returned would be truncated
   and the sender field was not returned. They are both now returned correctly.
   Bug report and partial fix provided by John Schultz. 
7) Fix bug where SP_Join and SP_Leave do not report an error if a group name
   is too long (instead they truncated it) Reported with fix by David Parker.
   ** Warning, this could break buggy applications who use long groups and 
      assume the name is truncated. 
8) Cleanup compile warnings where E_queue() used with no-parameter functions 
   (not all uses fixed) and fix incorrect use of signed int with strlen().
9) Fix few cases in flooder.c and user.c that did not use the defined MAX_MESSLEN
   constant. Tested to verify that increasing MAX_SCATTER_ELEMENTS in scatter.h
   and the MAX_MESSLEN defines in user.c and flooder.c is sufficient to support
   arbitrarily large message sizes with Spread. This is NOT recommended, but
   several people do it anyway :-)
10) Add new option to spflooder. The -n option allows a fixed "number of members"
    to be set. This then activates a multi-sender flow control algorithm to allow
    flooding tests with several senders. All of the processes need to join the
    group (i.e. they cannot be -wo (write-only)) but not all processes have to send.
    This allows easy testing with differing numbers of senders (just change how
    many spflooders start with -ro and how many do not).
11) Make E_delay() work on Win32. 
12) Added check that a segment using localhost (i.e. 127.0.0.255) is NOT in a 
    spread.conf with any other segments using real IP addresses. This configuration
    will not work correctly and is usually because the example localhost configuration
    provided with Spread is added to. This check will cause the daemon to exit 
    immediately after parsing the config file.
13) Add new Alarm priority flag to print a line with no datestamp (for multi-line output).
14) Fix bug where if more then 22 daemons start at the same time, some will crash or the 
    membership will not complete correctly. This bug was reported by several people 
    including Jesse Noller.  
15) Fix Java spread connection problem where several threaded connections from same 
    process cause unexpected connection failures. Reported by Brian Moseley, 
    idea of fix by Ryan Caudy, patch by Jonathan Stanton.
16) Fix EVS bug where AGREED messages may be delivered before a transitional signal
    on some daemons and after it on others. Bug found and patch created by 
    Ryan Caudy.
17) Make SP_connect_timeout() calls non-blocking for the actual 'connect()' call. 
    This should fix the issue reported by Shlomi Yaakobovich where a hung daemon
    causes new connections to also hang in connect. Also includes slight cleanup
    of connect code path.
18) Add new Windows VC++.Net project files to daemon/win32_msvc_net. The current
    project files in daemon/win32 remain as they can be used in VC++ (version 6).

SOURCE INSTALL:
---------------

The source install uses autoconf and make to support many Unix and unix-like 
platforms as well as Windows 95+.

Windows is supported by a set of Visual Studio desktop and project files in
the win32 subdirectory. These files build the daemon, libraries, and user
programs.

Any not-specifically supported
platform that has reasonably close to normal sockets networking should also 
be easily portable. See the file PORTING for hints on porting.

From the directory where you unpacked the Spread source distribution do 
the following:

1) Run "./configure"

2) Run "make"

3) Run "make install" as a user with rights to write to the selected
installation location. (/usr/local/{bin,man,sbin,include,lib} by default)

4) Create a spread.conf file appropriate to your local configuration. For 
more info on how to do this look at the sample file "sample.spread.conf"
or below in the binary install instructions.


BINARY INSTALL:
--------

We recommend that if you are experimenting with spread you create a special
'spread' directory (for example /usr/local/spread or /opt/spread) and keep
all the files together there so it is easy to find stuff.  That also makes 
it easier to run multiple architectures as the binaries for each are in their
own subdirectory. This is not necessary though. You can create that directory
anywhere (e.g. your own directory).

If you are installing spread for active use it is probably easier to just 
install the correct version of the binaries, headers, man pages, and 
libraries into your standard locations.  The directions below assume you 
are doing this.

1) Unpack the spread.tar.gz file into a temporary directory

2) Look at the Readme for any updates

3) Select the appropriate architecture:
	arch-bsdi
	arch-sgi
	arch-sunos
	arch-sunsol
	arch-pcsol
	arch-linux

4) Type 'make arch-????' with your architecture as the option to make

5) Now you need to copy the files, 
   I will assume you use /usr/local/{bin,include,lib,man}
   Replace "ARCH" with the directory for your architecure.

	cp -p include/* /usr/local/include/
	cp -p ARCH/libspread.a /usr/local/lib/
	cp -p ARCH/libtspread.a /usr/local/lib/
	cp -p ARCH/spread /usr/local/bin/
	cp -p ARCH/spmonitor /usr/local/bin/
	cp -p ARCH/spuser /usr/local/bin/
	cp -p ARCH/sptuser /usr/local/bin/
	cp -p ARCH/spflooder /usr/local/bin/
	cp -p docs/*.3 /usr/local/man/man3/
	cp -p docs/*.1 /usr/local/man/man1/

6) To run spread you need to modify the 'spread/sample.spread.conf' file for
   your network configuration.  Comments in the sample file explain this.
   Then cp -p sample.spread.conf /usr/local/bin/spread.conf
   (Notice the name change.  The config file must be named that to be found.
    Alternatively you can run spread -c <config_file_name> )

    You can run "spread usage" to see the spread running options.


To use the Java classes and examples you need to have a copy of the 
main 'spread' daemon running.  Then the sp.class file gives you the 
equivelent of the libsp.a as a java class.  The user.java, user.html, 
and user.class files give you a demonstration applet and source code.  
The tree.html AllNames.html and packages.html give some documentation 
for the java interface.

For Windows (95/NT) systems use the spread.exe daemon and the libsp.lib 
to link with your programs.


OVERVIEW:
---------

Spread consists of two parts, an executable daemon which is executed on 
each machine which is part of the heavyweight 'machine' group, and a library
 which provides a programming interface for writing group multicast 
applications and which is linked into the application.

	The daemon, called "spread", should be run as a non-priveledged 
user (we created a 'spread' user) and only needs permissions to create a 
socket in /tmp and read its config file which should be in the same 
directory as the executable.  By default the daemon binds to and runs 
off the non-priveledged port 4803 unless the config file indicates otherwise.
	Each daemon can be started independently and if it does not find 
any other members of its defined configuration currently active it runs as 
a machine group of 1 machine.  When other daemons are started they will join
this machine group.

	The machines which are running a daemon with a common config file 
form a 'machine group' (in contrast to a 'process group').  The daemons 
handle multicasting mesages between each other,providing ordering and 
delivery guarantees, detecting and handling failures of nodes or links, and 
managing all applications which are connected to each daemon.

	Each application utilizing the spread system needs to link with 
the 'libspread.a', 'libtspread.a', 'libtspread.lib' or 'libspread.lib' 
library and needs to use the calls defined in the 'sp.h' and 'sp_func.h' 
header files. Documentation for the interface is currenly incomplete, but 
a basic application must do at least the following:

1) Before using any other spread calls you need to 'connect' to a daemon 
   by calling the
	SP_connect(...) call.  This will set a mbox which is an integer 
representing your connection to the daemon.  You use this when making other
spread calls.  You CAN connect to daemons multiple times from the same 
application (and we know of times when this is very useful).

2) If you want to receive messages from a group you need to call SP_join().
Groups are named with standard alphanumeric strings.

3) To send to a group you do NOT need to join it, just call SP_multicast() 
with an appropriate group name.  You will not receive the messages back if 
you are not a member of the group.

4) To be nice to everyone else you should call SP_disconnect() when your 
program is finished using the spread system, if you do not your program's 
disconnection will eventually be detected. 

Some important observations when writing spread programs:

1) All the messages for a particular connection are received during an 
SP_receive() call.  This includes messages from different groups and 
membership messages(like group leave and join), so you must demultiplex 
them yourself.  This is a feature. You can opt to avoid getting membership 
messages all together by indicating that to SP_connect().

2) Spread handles endian differences correctly for metadata and our headers 
but does NOT convert your data for you.  It DOES tell you in the SP_receive 
call whether or not the data originated at machine with a different 
endianness then yours so you can easily use this to convert it yourself 
if necessary. 

3) Each connection to a daemon has a unique 'private group' name which 
can be used to send a message to a particular, and only to a particular, 
application connection.  

4) Spread supports the following ordering/delivery guarantees for messages:

	Unreliable	(least)
	Reliable	(will get there, no ordering)
	Fifo		(reliable and ordered fifo by source)
	Causal		(reliable and all mesg from any source of this level
			are causally ordered)
	Agreed		(reliable and all mesg from any source of this level 
			are totally ordered)
	Safe		(Agreed ordering and mesg will not be delivered to 
			application until the mesg has reached ALL 
			receipients' daemons)

For more detail, much of which IS important, see papers on 
Extended Virtual Sychrony, Transis and Totem.


UTILITY PROGRAMS:
-----------------

	Spread comes with several demonstration programs and useful tools.

1) spmonitor:
	This program has a special interface to spread which allows it to 
control a machine group.  It can terminate all the spread daemons in the 
group, change flow control parameters, and monitor the stats for one or all 
of the machines.  It needs to know the spread.conf file used for a particular
 set of spread daemons.

2) spuser:

	This program is provided with source and uses all the functions of 
the spread interface.  It also acts as a simple client which is useful for 
testing. Reading the source code to this program should show you how to use
all the features of spread, and answer many questions about syntax details.

3) simple_user:

	This is just about the simplest spread program possible and is 
provided with source.  It sends and receives text strings.

4) spflooder:
	
	This is used as one type of performace test.  It 'floods' a spread 
group with data messages. It is provided with source.


CONTACT:
--------

If you have any questions please feel free to contact:

	commedia@cnds.jhu.edu
	yairamir@cs.jhu.edu
	jonathan@cs.jhu.edu
