Segmentation fault with UDT file descriptors

January 24th, 2010 Danny Y. Huang No comments

When Gush is first started, segmentation fault appears. Here is the backtrace of the stack.

From the following line in TCPConnection::listen

loop->setHandlers(sock, _accept_handler, NULL, _accept_handler, true, false);

the program jumps to line #166 in event_loop_select.cc:

144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
void SelectEventLoop::setHandlers(int fd, ReadHandler *rh, WriteHandler *wh,
	ErrorHandler *eh, bool reading, bool writing)
{
    debugn(6, "setHandlers(" << fd << ", rh=" << (void *)rh << ", wh=" <<
	    (void *)wh << ", eh=" << (void *)eh << ", r=" << reading <<
	    ", w=" << writing << ")" );
    gush_assert( fd >= 0, "fd < 0 in setHandlers()");
 
    const unsigned int ufd = fd;
    MutexHolder lock_it(_lock);
 
    if (_read_handlers.size() <= ufd) {
	_read_handlers.resize(fd+1);
    }
    if (_write_handlers.size() <= ufd) {
	_write_handlers.resize(fd+1);
    }
    if (_error_handlers.size() <= ufd) {
	_error_handlers.resize(fd+1);
    }
    _read_handlers[fd] = rh;
    if ( reading ) {
	FD_SET( fd, &_read_fds );
    } else {
	FD_CLR( fd, &_read_fds );
    }
 
    _write_handlers[fd] = wh;
    if ( writing ) {
	FD_SET( fd, &_write_fds );
    } else {
	FD_CLR( fd, &_write_fds );
    }
 
    _error_handlers[fd] = eh;
    if ( eh != NULL ) {
	FD_SET( fd, &_error_fds );
    } else {
	FD_CLR( fd, &_error_fds );
    }
    updateMaxFD( fd );
}

Turns out that I need to change all FD_??? macros into UDT4’s very own UD_??? ones.

Categories: research Tags:

Statically compile the gush client

January 24th, 2010 Danny Y. Huang No comments
CLIENT_LDFLAGS = $(CFLAGS) $(LDFLAGS) $(STATIC_LDFLAGS)
CLIENT_LDFLAGS_POST = $(LDFLAGS_POST)

Use “ldd client” to check its dependency. If it is compiled statically, no dependencies will be displayed. Such a static compilation should solve the problem that the client cannot run on Planet-Lab machines.

Categories: research Tags:

XML-RPC Library not working on Planet-Lab

January 16th, 2010 Danny Y. Huang No comments

I modified the TCPConnection class to incorporate the UDT libraries. It was a pain to compile and link, but I suppose the server somehow worked—in a sense that the gush server can work without runtime errors. No changes were made in client.cc, but since it was statically compiled, the client executable has also changed. Yet when run on a PL machine, the client reported:

“./client: error while loading shared libraries: libxmlrpc_client++.so.3: cannot open shared object file: No such file or directory”

Since I was not able to run the client at all, there is no way to test for correctness. All I see right now is that gush can bootstrap but cannot make “TCP” connections (I didn’t change the interfaces yet). Installing the XML-RPC Library on PL is impossible because gcc is not available. Installing from the RPM yielded some wierd depency problem. Basically, I’ve been fighting the linking issue for a while now. I feel it is hard to proceed unless I can get the client to work.

Categories: research Tags:

UDT

January 16th, 2010 Danny Y. Huang No comments

Getting the UDT library to compile on Gush is a PAIN! More details coming up.

Categories: research Tags:

Gush without fcntl

January 13th, 2010 Danny Y. Huang No comments

I removed all the fcntl() statements from the TCPConnection class. Apparently, Gush worked fine, albeit slow at the start when I did “connect slice williams_kudzu”. I’m still rather confused about the purpose of fcntl(). If it is to set the bits for fds used in forking, apparently a totally new set of localized fds (like 0, 1, 2) are used in the multiprocess communications.

Important precondition: Prior to the removal of the fcntl() statements, Gush was run on a set of clean nodes. As a result, these nodes had already had the Gush clients by the time Gush was recompiled without the fcntl() statements. I don’t know what role this fact plays. Recorded here for future reference.

Anyways, it is good news. The fcntl() statements are the only socket-related calls that cannot be emulated in UDT. Since they’re not absolutely essential, I can leave them out in the UDTConnection class. Basically, the new UDTConnection class will be (ideally) somewhat similar to TCPConnection; all socket-related calls will be replaced by equivalent UDT calls (along with some nasty hacks just to get things to work).

This echos my earlier concern:

I have been looking for an application-layer protocol and I came across UDT (http://udt.sourceforge.net). Though its intended use is for high-speed networks, its potential to be used in the TCPConnection class is likely; it provides a similar interface to BSD sockets, and that it talks in UDP. Right now, I’m trying to replace BSD-socket statements into UDT-socket statements, e.g. from “int sock = std::socket(…)” into ” UDTSOCKET sock = UDT::socket(…)”. This is a hello-world application in UDT: http://udt.sourceforge.net/udt4/doc/t-hello.htm — it is not very different from a typical BSD-socket application.

Yet UDT is not the panacea. Some fd-related operations simply do not have their counterparts in the UDT world. For instance, “fcntl(sock, F_SETFD, FD_CLOEXEC)” and “fcntl(sock, F_SETFL, O_NONBLOCK)” are not possible if the UDT library is used. If I’m not wrong, they are meant for interprocess communications as proxy installers are spawned. Given that we still need fds to facilitate interproces communications, there is still a chance that Gush may run out of fds if processes are spawned too quickly, or if these processes don’t exit fast enough because of network latency (which is usually the case). These slow processes eat up the available fds, causing other new connections to starve. Since this is the main source where fds are consumed the most, I do not know how effective the TCP-to-UDT conversion may be with respect to the running-out-of-fd problem. Of course, adopting UDT has the potential of increased performance (or so the paper claims).

For now, I’m hacking the source code of UDT and trying to fit it into Gush. As for the “fcntl” cases, I may eventually leave them out in the UDT version and figure out an alternative.

Categories: research Tags:

Application layer transport protocols over UDP

January 7th, 2010 Danny Y. Huang No comments

See UDT: http://udt.sourceforge.net/ Looks very promising.

Categories: research Tags:

TCP over UDP literature

December 11th, 2009 Danny Y. Huang No comments

I haven’t found any articles on TCP over UDP at the application layer.

Will work on a small test program in c before hacking Gush.

Categories: research Tags:

Varying ulimit

December 11th, 2009 Danny Y. Huang No comments

I tried to vary the number of max fds to see the effects on the number of successful connections.

ulimit -n success
70 9
71 8*
72 10
73 11
74 13
75 13

* on second trial, 11 successful connections

The results of the tests, however, cannot be replicated over multiple trials. Apparently, each connection uses at least two file descriptors. One for the actual TCP connection, the other for the proxy (which essentially spawns an SSH and copies the client over). My guess is that the proxy is created too fast, such that they’re eating up too many file descriptors and leaving no fds for the TCP connections.

Sample output:

gush> connect slice williams_gush
Found 30 hosts
Initiated connections to 30 of 30 hosts.
gush> For an unknown reason, we were unable to invite williams_gush@planetlab1.ucsd.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab01.cs.washington.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab02.cs.washington.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab03.cs.washington.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab04.cs.washington.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab05.cs.washington.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab2.ucsd.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab3.ucsd.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab7.cs.duke.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab2.cs.duke.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab3-dsl.cs.cornell.edu:15001.
williams_gush@planetlab-04.cs.princeton.edu:15001 has joined the mesh.
williams_gush@planetlab-9.cs.princeton.edu:15001 has joined the mesh.
williams_gush@planetlab-02.cs.princeton.edu:15001 has joined the mesh.
williams_gush@planetlab5.cs.duke.edu:15001 has joined the mesh.
williams_gush@planetlab6.cs.cornell.edu:15001 has joined the mesh.
williams_gush@planetlab4-dsl.cs.cornell.edu:15001 has joined the mesh.
williams_gush@planetlab1.cs.cornell.edu:15001 has joined the mesh.
williams_gush@planetlab3.williams.edu:15001 has joined the mesh.
williams_gush@planet1.scs.stanford.edu:15001 has joined the mesh.
williams_gush@planet2.scs.stanford.edu:15001 has joined the mesh.
williams_gush@planetlab2.williams.edu:15001 has joined the mesh.
williams_gush@planetlab4.williams.edu:15001 has joined the mesh.
williams_gush@planetlab5.williams.edu:15001 has joined the mesh.
Categories: research Tags:

Unable to fork

December 10th, 2009 Danny Y. Huang No comments

Compare the two. One on the left shows the logfile for a successful connection, while the right one shows a failed connection. Test was carried out at ulimit -n 70. If you scroll, you’d probably notice that things had gone wrong from the blue line onwards.

1965:ssh_mesh.cc:674:SSHMesh::notifyOperationFailed(williams_gush@planetlab2.williams.edu:15001)
1966:ssh_mesh.cc:1526:inviteHandler(rhost=williams_gush@planetlab2.williams.edu:15001)
1967:ssh_mesh.cc:1553:  connecting to remote node
1968:ssh_mesh.cc:1587:  trying to start remote proxy… (tcp mode)
1969:proxy_installer.cc:503:arg[0] = /usr/bin/ssh
1970:proxy_installer.cc:503:arg[1] = -q
1971:proxy_installer.cc:503:arg[2] = -x
1972:proxy_installer.cc:503:arg[3] = -e
1973:proxy_installer.cc:503:arg[4] = none
1974:proxy_installer.cc:503:arg[5] = -l
1975:proxy_installer.cc:503:arg[6] = williams_gush
1976:proxy_installer.cc:503:arg[7] = planetlab2.williams.edu
1977:proxy_installer.cc:503:arg[8] = ./client
1978:proxy_installer.cc:503:arg[9] = -c
1979:proxy_installer.cc:503:arg[10] = ./
1980:proxy_installer.cc:503:arg[11] = -d
1981:proxy_installer.cc:503:arg[12] = 5
1982:proxy_installer.cc:503:arg[13] = -P
1983:proxy_installer.cc:503:arg[14] = 15001
1984:proxy_installer.cc:503:arg[15] = -b
1985:proxy_installer.cc:503:arg[16] = yh1@sysnet1:13456
1986:proxy_installer.cc:506:Starting proxy as williams_gush@planetlab2.williams.edu:15001
1987:process.cc:65:Process:Process(path=/usr/bin/ssh, args)
1988:process_monitor.cc:68:ProcessMonitor::attach:0xa0f5518, StartProxy on williams_gush@planetlab2.williams.edu:15001
1989:process.cc:638:Process::startNow
1990:process.cc:1324:ready to fork.
1991:process.cc:1452:back in parent
1992:process.cc:1473:parent process monitor stuff
1993:process.cc:1499:done in parent
1994:process.cc:1339:we’re forked! in child. hooking up.
1995:stream_input_channel.cc:161:StreamInputChannel::handleWrite()
1996:stream_input_channel.cc:120:StreamInputChannel::startPostForkChild
1997:connection_tcp.cc:345:Socket 35 has errors: 111, Connection refused
1998:connection_tcp.cc:361:Cleaning up socket 35
1999:stream_input_channel.cc:126:StreamInputChannel::startPostForkChild 6,0
2000:ssh_mesh.cc:674:SSHMesh::notifyOperationFailed(williams_gush@planetlab3.williams.edu:15001)
2001:stream_input_channel.cc:130:StreamInputChannel::startPostForkChild
2002:ssh_mesh.cc:1526:inviteHandler(rhost=williams_gush@planetlab3.williams.edu:15001)
2003:stream_input_channel.cc:135:StreamInputChannel::startPostForkChild
2004:ssh_mesh.cc:1553:  connecting to remote node
2005:stream_output_channel.cc:129:StreamOutputChannel::startPostForkChild
2006:stream_output_channel.cc:129:StreamOutputChannel::startPostForkChild
2007:ssh_mesh.cc:1587:  trying to start remote proxy… (tcp mode)
2008:process.cc:1395:   my_args[0]=/usr/bin/ssh
2009:process.cc:1395:   my_args[1]=-q
2351:ssh_mesh.cc:674:SSHMesh::notifyOperationFailed(williams_gush@planetlab01.cs.washington.edu:15001)
2352:ssh_mesh.cc:1526:inviteHandler(rhost=williams_gush@planetlab01.cs.washington.edu:15001)
2353:ssh_mesh.cc:1553:  connecting to remote node
2354:ssh_mesh.cc:1587:  trying to start remote proxy… (tcp mode)
2355:proxy_installer.cc:503:arg[0] = /usr/bin/ssh
2356:proxy_installer.cc:503:arg[1] = -q
2357:proxy_installer.cc:503:arg[2] = -x
2358:proxy_installer.cc:503:arg[3] = -e
2359:proxy_installer.cc:503:arg[4] = none
2360:proxy_installer.cc:503:arg[5] = -l
2361:proxy_installer.cc:503:arg[6] = williams_gush
2362:proxy_installer.cc:503:arg[7] = planetlab01.cs.washington.edu
2363:proxy_installer.cc:503:arg[8] = ./client
2364:proxy_installer.cc:503:arg[9] = -c
2365:proxy_installer.cc:503:arg[10] = ./
2366:proxy_installer.cc:503:arg[11] = -d
2367:proxy_installer.cc:503:arg[12] = 5
2368:proxy_installer.cc:503:arg[13] = -P
2369:proxy_installer.cc:503:arg[14] = 15001
2370:proxy_installer.cc:503:arg[15] = -b
2371:proxy_installer.cc:503:arg[16] = yh1@sysnet1:13456
2372:proxy_installer.cc:506:Starting proxy as williams_gush@planetlab01.cs.washington.edu:15001
2373:process.cc:65:Process:Process(path=/usr/bin/ssh, args)
2374:process_monitor.cc:68:ProcessMonitor::attach:0xa0f7c60, StartProxy on williams_gush@planetlab01.cs.washington.edu:15001
2375:process.cc:638:Process::startNow
2376:stream_input_channel.cc:78:Wert: Unable to pipe() in startPreFork:Too many open files
2377:process.cc:1306:Wert: File descriptor 0 SPF error: Too many open files
2378:exception.cc:27:Exception::Exception(Error while calling startPreFork().): this=0xa0f7fc8
2379:exception.cc:34:Exception::Exception(const Exception &0xa0f7fc8): this=0xbfc70450
2380:exception.cc:39:Exception::~Exception. this=0xbfc70450
2381:ssh_mesh.cc:1658:   -> exception!
2382:exception.cc:39:Exception::~Exception. this=0xa0f7fc8
2383:ssh_mesh.cc:1581:  connecting via TCP…
2384:connection_tcp.cc:151:connectTo(williams_gush@planetlab01.cs.washington.edu:15001)
2420:
2421:ssh_mesh.cc:609:SSHMesh::registerConnection(williams_gush@planetlab01.cs.washington.edu:15001)
2422:ssh_mesh.cc:610:Mesh registering connection williams_gush@planetlab01.cs.washington.edu:15001: ESC[33;01mconnecting via TCPESC[0m; (fds=-1,-1 last_heard=0)
2423:
2424:connection_tcp.cc:190:tcp_connection_handler:  hostname = williams_gush@planetlab01.cs.washington.edu:15001
2425:connection_tcp.cc:200:     setting up connection handler: fd=28
2426:connection_tcp.cc:345:Socket 29 has errors: 111, Connection refused
2427:connection_tcp.cc:361:Cleaning up socket 29
2428:ssh_mesh.cc:674:SSHMesh::notifyOperationFailed(williams_gush@planetlab02.cs.washington.edu:15001)
2429:ssh_mesh.cc:1526:inviteHandler(rhost=williams_gush@planetlab02.cs.washington.edu:15001)
2430:ssh_mesh.cc:1553:  connecting to remote node

The root of the problem is the pipe() system call. Apparently it cannot create any more fds. Here is stream_input_channel.cc:

int StreamInputChannel::startPreFork(void)
{
    MutexHolder lock_it( _lock );
    gush_assert(_process != NULL, "Process for StreamInputChannel not set!");
    gush_assert(_fd >= 0, "File descriptor for StreamInputChannel not set!");
 
    int err = pipe(_pipe);
    if (err < 0) {
        debugn(1, "Wert: Unable to pipe() in startPreFork:"
                  << strerror(errno));
        debugn(1, "zzzz Pipe screwed. _pipe[0] = " << _pipe[0]
               << " and _pipe[1] = " << _pipe[1] << ".");
        return err;
    }
 
    return 0;
}

And this is the debug print of the “zzzz” line:

stream_input_channel.cc:80:zzzz Pipe screwed. _pipe[0] = -1 and _pipe[1] = -1.
Categories: research Tags:

Gush screwed at ulimit -n 70

December 10th, 2009 Danny Y. Huang No comments
yh1@sysnet1:~$ ulimit -n 70
yh1@sysnet1:~$ cd gush
yh1@sysnet1:~/gush$ ./gush -P 13456
gush> Gush has learned about the slice williams_gush.
connect slice williams_gush
Found 30 hosts
Initiated connections to 30 of 30 hosts.
gush> For an unknown reason, we were unable to invite williams_gush@planet1.scs.stanford.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planet2.scs.stanford.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab01.cs.washington.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab02.cs.washington.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab03.cs.washington.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab04.cs.washington.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab05.cs.washington.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab1.ucsd.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab2.ucsd.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab3.ucsd.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab6.cs.duke.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab7.cs.duke.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab2.cs.duke.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab4.cs.duke.edu:15001.
For an unknown reason, we were unable to invite williams_gush@planetlab3-dsl.cs.cornell.edu:15001.
williams_gush@planetlab-04.cs.princeton.edu:15001 has joined the mesh.
williams_gush@planetlab-02.cs.princeton.edu:15001 has joined the mesh.
williams_gush@planetlab-9.cs.princeton.edu:15001 has joined the mesh.
williams_gush@planetlab5.williams.edu:15001 has joined the mesh.
williams_gush@planetlab6.cs.cornell.edu:15001 has joined the mesh.
williams_gush@planetlab3.williams.edu:15001 has joined the mesh.
williams_gush@planetlab1.cs.cornell.edu:15001 has joined the mesh.
williams_gush@planetlab4.williams.edu:15001 has joined the mesh.
williams_gush@planetlab2.williams.edu:15001 has joined the mesh.
williams_gush@planetlab4-dsl.cs.cornell.edu:15001 has joined the mesh.
 
gush> info mesh
Mesh:
williams_gush@planetlab1.cs.cornell.edu:15001: connected; (fds=11,11 last_heard=1260495974)
williams_gush@planetlab3-dsl.cs.cornell.edu:15001: failed; (fds=-1,-1 last_heard=0)
williams_gush@planetlab4-dsl.cs.cornell.edu:15001: connected; (fds=9,9 last_heard=1260495975)
williams_gush@planetlab6.cs.cornell.edu:15001: connected; (fds=20,20 last_heard=1260495974)
williams_gush@planetlab1.cs.duke.edu:15001: connecting via TCP; (fds=-1,-1 last_heard=0)
williams_gush@planetlab2.cs.duke.edu:15001: failed; (fds=-1,-1 last_heard=0)
williams_gush@planetlab3.cs.duke.edu:15001: connecting via TCP; (fds=-1,-1 last_heard=0)
williams_gush@planetlab4.cs.duke.edu:15001: failed; (fds=-1,-1 last_heard=0)
williams_gush@planetlab5.cs.duke.edu:15001: connecting via TCP; (fds=-1,-1 last_heard=0)
williams_gush@planetlab6.cs.duke.edu:15001: failed; (fds=-1,-1 last_heard=0)
williams_gush@planetlab7.cs.duke.edu:15001: failed; (fds=-1,-1 last_heard=0)
williams_gush@alice.cs.princeton.edu:15001: connecting via TCP; (fds=-1,-1 last_heard=0)
williams_gush@planetlab-02.cs.princeton.edu:15001: connected; (fds=38,38 last_heard=1260495974)
williams_gush@planetlab-04.cs.princeton.edu:15001: connected; (fds=21,21 last_heard=1260495974)
williams_gush@planetlab-9.cs.princeton.edu:15001: connected; (fds=22,22 last_heard=1260495974)
williams_gush@planet1.scs.stanford.edu:15001: failed; (fds=-1,-1 last_heard=0)
williams_gush@planet2.scs.stanford.edu:15001: failed; (fds=-1,-1 last_heard=0)
williams_gush@planetlab1.ucsd.edu:15001: failed; (fds=-1,-1 last_heard=0)
williams_gush@planetlab2.ucsd.edu:15001: failed; (fds=-1,-1 last_heard=0)
williams_gush@planetlab3.ucsd.edu:15001: failed; (fds=-1,-1 last_heard=0)
williams_gush@planetlab01.cs.washington.edu:15001: failed; (fds=-1,-1 last_heard=0)
williams_gush@planetlab02.cs.washington.edu:15001: failed; (fds=-1,-1 last_heard=0)
williams_gush@planetlab03.cs.washington.edu:15001: failed; (fds=-1,-1 last_heard=0)
williams_gush@planetlab04.cs.washington.edu:15001: failed; (fds=-1,-1 last_heard=0)
williams_gush@planetlab05.cs.washington.edu:15001: failed; (fds=-1,-1 last_heard=0)
williams_gush@planetlab06.cs.washington.edu:15001: connecting via TCP; (fds=-1,-1 last_heard=0)
williams_gush@planetlab2.williams.edu:15001: connected; (fds=34,34 last_heard=1260495974)
williams_gush@planetlab3.williams.edu:15001: connected; (fds=35,35 last_heard=1260495974)
williams_gush@planetlab4.williams.edu:15001: connected; (fds=36,36 last_heard=1260495974)
williams_gush@planetlab5.williams.edu:15001: connected; (fds=37,37 last_heard=1260495974)
Summary: 10 mesh members; 5 nodes connecting.
gush>

Here is the FD Map:

7242:connection_tcp.cc:43:zzzz FD MAP zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
7243:connection_tcp.cc:44:zzzz size = 35
7244:connection_tcp.cc:47:zzzz    6	connectTo(): williams_gush@planetlab1.cs.cornell.edu:15001
7245:connection_tcp.cc:47:zzzz    7	connectTo(): williams_gush@planetlab3-dsl.cs.cornell.edu:15001
7246:connection_tcp.cc:47:zzzz    9	connectTo(): williams_gush@planetlab4-dsl.cs.cornell.edu:15001
7247:connection_tcp.cc:47:zzzz    10	listen()
7248:connection_tcp.cc:47:zzzz    11	connectTo(): williams_gush@planetlab1.cs.cornell.edu:15001
7249:connection_tcp.cc:47:zzzz    12	connectTo(): williams_gush@planetlab1.cs.duke.edu:15001
7250:connection_tcp.cc:47:zzzz    13	connectTo(): williams_gush@planetlab2.cs.duke.edu:15001
7251:connection_tcp.cc:47:zzzz    14	connectTo(): williams_gush@planetlab3.cs.duke.edu:15001
7252:connection_tcp.cc:47:zzzz    15	connectTo(): williams_gush@planetlab4.cs.duke.edu:15001
7253:connection_tcp.cc:47:zzzz    16	connectTo(): williams_gush@planetlab5.cs.duke.edu:15001
7254:connection_tcp.cc:47:zzzz    17	connectTo(): williams_gush@planetlab6.cs.duke.edu:15001
7255:connection_tcp.cc:47:zzzz    18	connectTo(): williams_gush@planetlab7.cs.duke.edu:15001
7256:connection_tcp.cc:47:zzzz    19	connectTo(): williams_gush@alice.cs.princeton.edu:15001
7257:connection_tcp.cc:47:zzzz    20	connectTo(): williams_gush@planetlab6.cs.cornell.edu:15001
7258:connection_tcp.cc:47:zzzz    21	connectTo(): williams_gush@planetlab-04.cs.princeton.edu:15001
7259:connection_tcp.cc:47:zzzz    22	connectTo(): williams_gush@planetlab-9.cs.princeton.edu:15001
7260:connection_tcp.cc:47:zzzz    23	connectTo(): williams_gush@planet1.scs.stanford.edu:15001
7261:connection_tcp.cc:47:zzzz    24	connectTo(): williams_gush@planet2.scs.stanford.edu:15001
7262:connection_tcp.cc:47:zzzz    25	connectTo(): williams_gush@planetlab1.ucsd.edu:15001
7263:connection_tcp.cc:47:zzzz    26	connectTo(): williams_gush@planetlab2.ucsd.edu:15001
7264:connection_tcp.cc:47:zzzz    27	connectTo(): williams_gush@planetlab3.ucsd.edu:15001
7265:connection_tcp.cc:47:zzzz    28	connectTo(): williams_gush@planetlab01.cs.washington.edu:15001
7266:connection_tcp.cc:47:zzzz    29	connectTo(): williams_gush@planetlab02.cs.washington.edu:15001
7267:connection_tcp.cc:47:zzzz    30	connectTo(): williams_gush@planetlab03.cs.washington.edu:15001
7268:connection_tcp.cc:47:zzzz    31	connectTo(): williams_gush@planetlab04.cs.washington.edu:15001
7269:connection_tcp.cc:47:zzzz    32	connectTo(): williams_gush@planetlab05.cs.washington.edu:15001
7270:connection_tcp.cc:47:zzzz    33	connectTo(): williams_gush@planetlab06.cs.washington.edu:15001
7271:connection_tcp.cc:47:zzzz    34	connectTo(): williams_gush@planetlab2.williams.edu:15001
7272:connection_tcp.cc:47:zzzz    35	connectTo(): williams_gush@planetlab3.williams.edu:15001
7273:connection_tcp.cc:47:zzzz    36	connectTo(): williams_gush@planetlab4.williams.edu:15001
7274:connection_tcp.cc:47:zzzz    37	connectTo(): williams_gush@planetlab5.williams.edu:15001
7275:connection_tcp.cc:47:zzzz    38	connectTo(): williams_gush@planetlab-02.cs.princeton.edu:15001
7276:connection_tcp.cc:47:zzzz    53	connectTo(): williams_gush@planetlab4.cs.duke.edu:15001
7277:connection_tcp.cc:47:zzzz    55	connectTo(): williams_gush@planetlab7.cs.duke.edu:15001
7278:connection_tcp.cc:47:zzzz    69	connectTo(): williams_gush@planet1.scs.stanford.edu:15001
7279:connection_tcp.cc:49:zzzz ......................................
Categories: research Tags: