Emulab generates an emulab.pem file which is by default encrypted with a PEM passphrase. It consists of two sections: the RSA private key (encrypted) and the certificate. We need to remove the PEM passphrase from the RSA private key for the file to work correctly in Gush.

Here is an example of what emulab.pem looks like:

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,552F11FEB2C33DB6
 
KGO5Rj9QcaRD6dfiuTHWnvAcwhER2yknQjUFTkB5/zz1JqMChAkH3q6nwpCehOVF
....
aUzRSY3KpBSCoKjpe0dcZkTHPoRqyCmcU75gioSYtbD2wWcvhGFsPA==
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
MIIDADCCAmkMAiRtMA0GCSqGSIb3DQEBBAUAMIG4MQswCQYDVQQGEwJVUzENMAsG
...
i6vWkw==
-----END CERTIFICATE-----

Before we proceed, please backup the original emulab.pem.

Open emulab.pem in a text editor and copy the certificate section into a new file called emulab.crt, which looks like this:

-----BEGIN CERTIFICATE-----
MIIDADCCAmkMAiRtMA0GCSqGSIb3DQEBBAUAMIG4MQswCQYDVQQGEwJVUzENMAsG
...
i6vWkw==
-----END CERTIFICATE-----

Next, we need to strip the RSA private key off its passphrase. In the terminal, run

openssl rsa -in emulab.pem -out rsa.key

Now that rsa.key contains no passphrase, we need to combine it with emulab.crt into a new Emulab SSL certificate.

cat rsa.key emulab.crt > emulab.pem

Finally, the passphrase-free emulab.pem can pose a security risk. This is the least we can do:

chmod 400 emulab.pem

This command obtains all nodes for the experiment defined:

pig script_wrapper.py --login=yh1 expinfo -n -e Gush,gush

You need to manually swap in/out experiments because it takes too much user-waiting time if the command line tool (script_wrapper.py) is used.

If this error occurs to your Keychain,

KeyChain 2.6.8; http://www.gentoo.org/proj/en/keychain/
Copyright 2002-2004 Gentoo Foundation; Distributed under the GPL
 
 * Found existing ssh-agent (3432)
 * Warning: can't find /home/yh1/.ssh/id_dsa; skipping
 * Adding 1 ssh key(s)...
Could not open a connection to your authentication agent.
 * Error: Problem adding; giving up

something is wrong with ssh-add.

Simple solution: restart your machine.

To disable root access via SSH, do the following. In this example, we use the vi editor. Replace it with the editor of your choice.

sudo vi /etc/ssh/sshd_config

Change this line:

#PermitRootLogin yes

to this:

#PermitRootLogin no

Finally, restart the ssh server. On Ubuntu, it’s

sudo /etc/init.d/ssh restart

But if you cannot find the ssh server, try locate sshd. It has a similar command-line interface.

Using the GeniWrapper for Gush ¶

Though not officially in the main SVN trunk, the GeniWrapper interface is currently integrated into Gush. In addition to communicating with the central database in the traditional PLC API, Gush can now talk in the GeniWrapper lingo.

Assumptions on system configurations ¶

For this example, the user name is jeannie [at] cs [dot] williams [dot] edu. Its Human-Readable Name (HRN) is plc.williams.jeannie. The slice I’m working with is williams_gush. I’m running Python 2.6 on Ubuntu 9.04. The latest version of Gush has been checked out and it was correctly configured as described in UsingGush.

Here are all the python packages you need to install:

$ sudo apt-get install python-m2crypto

Note that this is not a comprehensive list. If, for instance, the Python interpreter complains that some package XYZ is not found, run sudo apt-get install python-XYZ to install the required packages. Usually this solves the missing-package problem.

Setting up GeniWrapper for Gush ¶

First, create your private key for planetlab. In your ~/.ssh/ directory, run ssh-keygen -t rsa to generate your keypairs, id_rsa (your private key) and id_rsa.pub (your public key). It doesn’t matter whether a pass phrase is specified. Upload your public key to planetlab.

Copy your private key to the directory gush/helper-scripts/sfa/.sfi/ and rename it as jeannie.pkey. Edit the configuration file sfi_config in this directory. My sfi_config file looks like this:

SFI_AUTH='plc.williams'
SFI_USER='plc.williams.jeannie'
SFI_REGISTRY='http://www.planet-lab.org:12345/'
SFI_SM='http://www.planet-lab.org:12347/'

It is possible that you may have multiple configuration files for various types of resources. You can name them differently. Note that all of them need to be in the directory gush/helper-scripts/sfa/.sfi/.

In my case, for instance, I have williams_sfi_config which looks like this:

SFI_AUTH='plc.williams'
SFI_USER='plc.williams.jeannie'
SFI_REGISTRY='http://www.planet-lab.org:12345/'
SFI_SM='http://www.planet-lab.org:12347/'

and ksu_sfi_config that reads:

SFI_AUTH='plc.ksu'
SFI_USER='plc.ksu.jeannie'
SFI_REGISTRY='http://geni-myplc.ksu.gpeni.net:12345/'
SFI_SM='http://geni-myplc.ksu.gpeni.net:12346/'

Then, edit your gush/directory.xml file to specify resources for the GENI interface. If a resource uses the old PLC API, indicate the type as “planetlab”; to use the new GENI interface, specify “geni” as the type. Also, specify the name of the configuration file for a particular GENI resource. An example would be:

<?xml version="1.0" encoding="UTF-8"?>
<gush>
    <resource_manager type="planetlab">
        <user>yh1@williams.edu</user>
        <allsites>allsites.xml</allsites>
        <port_map slice="williams_gush" port="15000"/>
    </resource_manager>

    <resource_manager type="geni">
        <user>plc.williams.jeannie</user>
	<config_file>williams_sfi_config</config_file>
        <port_map slice="williams_gush" port="15000"/>
        <port_map slice="williams_kudzu" port="15000"/>
    </resource_manager>

    <resource_manager type="geni">
        <user>plc.ksu.jeannie</user>
	<config_file>ksu_sfi_config</config_file>
        <port_map slice="williams_gush" port="15000"/>
    </resource_manager>

    <resource_manager type="ssh">
    </resource_manager>
</gush>

To test whether you have configured correct, run the following command in the gush/ directory.

$ helper-scripts/handle-geniplcdb.py --noemail listall

The --noemail switch is included to speed up the process, so that the Python script will not translate the HRNs of each user into the corresponding email addresses. Try the --debug switch if you wish to see debug statements. Messy print-outs expected.

If all goes well, you should see something that looks like the following. The user names are blank because of the --noemail switch. This does not affect the performance of Gush.

$ helper-scripts/handle-geniplcdb.py --noemail listall 

<?xml version="1.0"?>
<gush>
	<slice name="williams_gush">
		<expires>1249326806</expires>
		<user name=""/>
		<user name=""/>
		<user name=""/>
		<node name="planetlab2.ucsd.edu"/>
		<node name="planetlab3.williams.edu"/>
		<node name="planetlab4.williams.edu"/>
		<node name="planetlab2.williams.edu"/>
		<node name="planetlab1.ucsd.edu"/>
		<node name="planetlab5.williams.edu"/>
		<node name="planetlab3.ucsd.edu"/>
		<node name="planetlab1.williams.edu"/>
	</slice>
</gush>

$

Running Gush with GeniWrapper ¶

By default, Gush polls the list of resources by communicating in PLC API with the planetlab server. This is achieved through the helper-scripts/handle-plcdb.pl script. To replace it with the new GeniWrapper, run Gush with the following command:

$ ./gush -P 15890 --geni

The --geni switch (or -g for short) tells Gush to obtain the list of resources with the new GeniWrapper interface. Currently, it takes some patience for Gush to learn about your slices. When it’s done, you should see this:

$ ./gush -P 15890 -d 0 --geni
Using GeniWrapper.
Gush has learned about the slice williams_gush.
gush>

At this point, Gush is no different from that which uses the old PLC API. You can run commands and carry out experiments as you normally would. If you run info nodes, you’ll find the correct list of nodes.

gush> info nodes
There are 8 known nodes:
[ P         ] williams_gush@planetlab1.ucsd.edu:15000(pref=0) (Disconnected.)
[ P         ] williams_gush@planetlab2.ucsd.edu:15000(pref=0) (Disconnected.)
[ P         ] williams_gush@planetlab3.ucsd.edu:15000(pref=0) (Disconnected.)
[ P         ] williams_gush@planetlab1.williams.edu:15000(pref=0) (Disconnected.)
[ P         ] williams_gush@planetlab2.williams.edu:15000(pref=0) (Disconnected.)
[ P         ] williams_gush@planetlab3.williams.edu:15000(pref=0) (Disconnected.)
[ P         ] williams_gush@planetlab4.williams.edu:15000(pref=0) (Disconnected.)
[ P         ] williams_gush@planetlab5.williams.edu:15000(pref=0) (Disconnected.)
gush>

The sfi.py utility still fails to authenticate.

Here’s the string that is sent to the RPC server for authentication. The difference is in Line 10.

In Jeannie’s case, it’s:

1
2
3
4
5
6
7
8
9
10
11
12
13

<methodCall>
<methodName>get_credential</methodName>
<params>
<param>
<value><nil/></value></param>
<param>
<value><string>user</string></value>
</param>
<param>
<value><string>plc.williams.jeannie</string></value>
</param>
</params>
</methodCall>

And in my case:

1
2
3
4
5
6
7
8
9
10
11
12
13

<methodCall>
<methodName>get_credential</methodName>
<params>
<param>
<value><nil/></value></param>
<param>
<value><string>user</string></value>
</param>
<param>
<value><string>plc.williams.yh1</string></value>
</param>
</params>
</methodCall>

Mine still fails. I’m clueless. For now, I’ll use Jeannie’s credentials and try to move on, integrating sfi.py into Gush.

To work on this next week: default_matcher.cc in Gush.

1
2
3
4
5
6
7
8
9

        printf("zzzz minNumHosts = %d\n", rspec->getMinNumHosts());
        // zzzz Commented out the following                                                                                                          
        // int remaining = rspec->getNumHosts() - m->getNumAssignments(config->getName());                                                           
 
        // zzzz Replaced with the following. If min num of host is not defined                                                                       
	// in XML, it is assigned the number of hosts.                                                                                               
 
        int remaining = rspec->getMinNumHosts() - m->getNumAssignments(config->getName());
        int surplus = rspec->getMaxNumHosts() - rspec->getNumHosts();

I’m getting Internal Server Errors for SFA/SFI/GeniWrapper, while in Raven/Stork the servers are not up. Oh God.

This is what happens in SFA/SFI:

yh1@sysnet1:~/.sfi$ source sfi_config
yh1@sysnet1:~/.sfi$ sfi.py list plc
Enter PEM pass phrase:
Traceback (most recent call last):
  File "/usr/local/bin/sfi.py", line 714, in <module>
    main()
  File "/usr/local/bin/sfi.py", line 382, in main
    dispatch(command, cmd_opts, cmd_args)
  File "/usr/local/bin/sfi.py", line 348, in dispatch
    globals()[command](cmd_opts, cmd_args)
  File "/usr/local/bin/sfi.py", line 399, in list
    user_cred = get_user_cred()
  File "/usr/local/bin/sfi.py", line 150, in get_user_cred
    user_cred = registry.get_credential(None, "user", user)
  File "/usr/local/lib/python2.6/dist-packages/sfa/util/geniclient.py", line 160, in get_credential
    cred_str = self.server.get_credential(cred, type, name)
  File "/usr/lib/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.6/xmlrpclib.py", line 1243, in request
    headers
xmlrpclib.ProtocolError: <ProtocolError for 128.112.139.90:12345/: 500 Internal Server Error>
yh1@sysnet1:~/.sfi$

Quote from Larry Peterson:

… multiple developers have their hand in the code, trying to work for some demo scenarios, so things will be a bit unstable for the next few days…

And for the Stork Utility, the application is trying to authenticate my login information, but it suspends forever while it is attempting to run this shell command:

/usr/local/bin/curl -L -k -d "username=yh1%40williams.edu&password=xxxxxxxx&authmethod=PLauthenticate" -D GetCookie https://stork-repository.cs.arizona.edu:8081/stork/login.php?

It was a good thing my hard disk was wiped out a few days ago. I got to re-install Raven/Stork. I followed exactly the same steps (here) but on the Raven/Stork website the Stork client was shown to be absent in my nodes. Very frustrating.

And the Stork GUI is problematic too. First it uses a deprecated package os.popen3. Next it tries to authenticate with a dysfunctional server: https://stork-repository.cs.arizona.edu:8081/stork/login.php.

Grounded.

I’ve been trying to break Gush. To be precise, break the internal connections of Gush to see how it does failure recovery. From here, I will try to implement something that looks like this:

<num_hosts min="1" max="100">8</num_hosts>

To achieve this goal, first I need to see how Gush determines the number of hosts. I placed a breakpoint at the

unsigned int ResourceSpecification::getNumHosts(void) const throw()

function in resources.cc. Gush was then executed in gdb. An experiment was loaded and ran. Whenever the breakpoint was triggered, I did a back trace to determine the chain of reaction.

Apparently, the following methods are worth close examinations:

yh1@sysnet1:~/gush$ grep getNumHosts *.cc
component_block.cc:	    ss << "Component requests " << rspec->getNumHosts() << " hosts.\n";
component_block.cc:	    ss << "Component requests " << rspec->getNumHosts() << " hosts.\n";
component_block.cc:    debug("done = " << done << " total = " << total << " numhosts = " << _component->getNumHosts());
component.cc:unsigned int Component::getNumHosts(void) const {
component.cc:    else return _rspec->getNumHosts();
container_block.cc:// 	    _restart_sync_barrier->setMax(_configuration->getRSpec()->getNumHosts());
default_matcher.cc:	int remaining = rspec->getNumHosts() - m->getNumAssignments(config->getName());
default_matcher.cc:	int surplus = rspec->getMaxNumHosts() - rspec->getNumHosts();
experiment_block.cc:// 	    _restart_sync_barrier->setMax(_configuration->getRSpec()->getNumHosts());
resources.cc:unsigned int ResourceSpecification::getNumHosts(void) const throw()
shirako_client.cc:    params.add(value_int(getNumHosts()));
shirako_client.cc:    structData.insert( make_pair("units.min", value_int(getNumHosts())) );
shirako_client.cc:    structData.insert( make_pair("units.max", value_int(getNumHosts())) );
sword_matcher.cc:		    BAD_CAST XMLUtil::itoa(comp->getNumHosts()).c_str());
sword_objects.cc:// 		    BAD_CAST XMLUtil::itoa(rspec->getNumHosts()).c_str());
workflow_block.cc:        rspec->setNumHosts( rspec->getNumHosts()+temp_failed_hosts.size() );
xmlrpc_callback.cc:    if (g_gush->getShirakoClient()->getNumHosts() == counter) {

I don’t know what DefaultMatcher does, so I’m trying to figure it out by disconnection nodes during experiments.

Nodes can be disconnected by killing the Gush client. This simple Python function can be called at fixed intervals so that no Gush client can survive:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

def kill():
    results = run('ps -eo "%p %c %a"').split("\n")
    regex = r"(\d+)\s+\w+"
    pList = []
    for item in results:
        if item.find("gush") != -1:
            print item
            r = re.search(regex, item)
            if r:
                pList.append(r.group(1))
 
    for item in pList:
        cmd = "kill -9 %s" % item
        print cmd
        print run(cmd)

I understand a shell script can handle the job more efficiently because new processes are created within this Python script. But to hell with shell script; it’s so ugly.

This is killing from within; next I’ll explain killing from the outside.

On sysnet1, I installed Firestarter, a simple Linux firewall. I first opened the ports for SSH and HTTP.

Open ports for SSH and HTTP

Then, I blocked connections to planetlab4.williams.edu.

Blocking planetlab4.williams.edu

I wrote a simple experiment that takes one minute to run on each node. Then halfway through the experiment, I turned on the firewall, or started the Python script which kills Gush clients on planetlab4. And I observed.