1. How do I sign up for the TestRig 2.0 service?
The first step is creating an account at testrig.psc.edu. Aside from a username, password, and contact information you’ll also need to provide us with the following:
- The hostname or ip address of the host that will receive the test data. On this host you’ll need to create an account to receive the data through scp. This account should be configured so that it does not provide an interactive shell on login. This is most easily accomplished by using rssh which is available as a package via yum/dnf or apt on most linux distributions or from http://www.pizzashack.org/rssh/. TestRig 2.0 does not work with the scponly application.
- The absolute path to where you would like the data to be stored on your system. For example: ‘/home/testrigdata/incoming’
- Optionally if you are using a trouble ticket system, such as RT, TestRig can be integrated automatically by providing the appropriate email address. For example: ‘rt-support@example.com’
After you have created the account, the TestRig team will review and validate the account within one business day. You will also be mailed an OpenSSH public key. This key is generated by our services and used to authenticate the transfer of the test results back to your local host. You will need to make sure that the target account on the local host is properly configured to make sue of this public key. In order to do this enter the target account and create a ./ssh directory. In that directory copy the public key into a file named ‘authorized_keys’. You will be able to generate new public keys as necessary through the administrative interface of your account on testrig.psc.edu.
2. Do I need to install any specific software on my servers?
TestRig 2.0 is provided as a Software as a Service offering to the larger community. As such, aside from software used to lock down the incoming results account mentioned above, there is no need to install additional TestRig 2.0 software. All communications, management, and authentication take place through a central TestRig 2.0 management server located at the Pittsburgh Supercomputing Center (PSC). This server creates and manages accounts, handles requests for new ISOs, generates these ISOs, provides authentication for the ISOs, and tracks usage.
3. What is the projected workflow for making use of TestRig 2.0?
When developing TestRig 2.0 the team developed a workflow which we feel will work with the large majority of customers. While this may not suit every customer we believe that it provided the most straightforward method of providing this service.
- Organization or customer registers an account at testrig.psc.edu
- Public key is sent to organization.
- Information is reviewed by PSC and account approved.
- Organization sets up incoming test results account on local resources.
- Account is secured with rssh
- Provided public key is added to ./ssh/authorized_keys file
- Organization receives user report of poor network performance
- Trouble ticket assigned
- Organization logs into TestRig 2.0 account and requests an ISO specific to that user.
- TestRig 2.0 server generates new ISO
- Trouble ticket updated by TestRig 2.0 (if applicable)
- User and Organization contacted when ISO is available
- User retrieves ISO from testrig.psc.edu
- Trouble ticket updated by TestRig 2.0 to indicate user has downloaded the ISO
- User creates bootable media from ISO
- User boots machine into TestRig 2.0 environment
- Tests are automatically conducted by TestRig 2.0 ISO
- Results are automatically returned to incoming test results account
- Trouble ticket updated to reflect that tests have been completed
- Organization contact also informed.
- Organization reviews data, determines likely cause of performance problem, and submits suggested fix to user.
4. How do I request a TestRig 2.0 ISO?
After logging into your account click the “Generate New ISO” tab at the top of the browser window. You’ll then need provide the following information
- The hostname/ip address of the end user’s machine.
- This is used to automatically determine the most appropriate perfSONAR node to run the tests against. If you leave this field blank you will need to manually choose a perfSONAR node.
- The maximum number of runs. Default of 7.
- How long the ISO is valid for. Default of 7 days.
- The associated trouble ticket number corresponding to the end user’s initial problem report. This field may be left blank.
- The end user’s name. This field may be left blank.
- The end user’s email address. This is how we will let them know that their ISO is ready for pickup. This is required information.
- Their affiliation; the organization that the end user is associated with. This field may be left blank.
- The RT queue name. If your trouble ticket system uses queues please enter the queue name associated with the end user’s request here. This field may be left blank.
- The tests that you would like to run. Currently the following tests are available:
- iperf
- iperf3
- nuttcp
- owping (one way ping measurement)
- ping
- tcpdump - a 30 second capture.
- tracepath
- traceroute (includes traceroute and paris-traceroute)
- UDP tests
Once you have submitted this information a new ISO will be automatically generated. After completion both the end user and account contact (you) will be informed that the ISO is ready for pickup. ISOs will remain available on the server for 4 days.
It’s important to note that TestRig 2.0 depends on DHCP for addressing and nameservice. The end user’s host must be in a network that can properly respond to DHCP requests.
5. My end user is on a machine with multiple interfaces. How can I tell TestRig which interface to use?
At this time this TestRig does not support multiple interfaces. The capability may be developed in the future. You may still use TestRig on multiply homed machines but only one interface will be tested. Which interface that is depends on how DHCP sets up default routes during boot.
6. Can I request that TestRig incorporate a new test?
All requests for new tests and/or modifications to existing tests will be considered by the TestRig 2.0. We are always on the lookout for new ideas.
7. How do I know the end user has gotten the ISO or has run TestRig?
The account contact and, if available, trouble ticket system will receive email once the user picks up the TestRig ISO and upon successfully completing the tests. You can then retrieve the data set from the data host defined during account setup.
8. How can the end users make bootable media from the ISO? Will you provide support for this?
End users have two options for creating bootable media. The first is by burning a CD or DVD with the ISO image. Mac OS X and Windows both have built in utilities to burn a CD from an ISO. Mac OS X users can use Disk Utility while Windows users (starting with Windows 7) simply need to insert blank media into a drive, right click on the ISO, a select Burn Image. The second is by creating a bootable flash drive. For systems without optical drives this would be the only feasible option. This process tends to be more complicated and requires the use of a helper application such as unetbootin (http://unetbootin.github.io/). The TestRig 2.0 distribution will have more extensive documentation on both methods but we cannot provide direct end user support.
9. Can my users run the ISO as many times as they want?
No. When you have a TestRig 2.0 ISO generated you can specify the number of times the ISO can be used, how many days it can be used, or both. The default settings are to allow an ISO to be used seven time over seven days. This is to prevent a user from using an ISO outside of the context of a reported problem. After the ISO has been run the maximum number of time or has expired the ISO will no longer be able to authenticate against our servers and will not conduct any tests.
10. How is an ISO authenticated?
When a TestRig 2.0 ISO is generated we create a unique public/private key pair for it. This key pair is used to encrypt a short string of randomly generated text. When the ISO boots it presents the encrypted known text to our authentication server which uses the corresponding private key to decrypt it. If that decrypted known text matches the known text stored on our servers the ISO is authenticated and unlocked. Without authentication the ISO is essentially unusable. This is designed into the system on purpose and gives us the ability to restrict the use of ISOs if we discover security issues, upgrade the underlying software, and so forth allowing us to ensure the same baseline environment across all ISOs.
11. I’m going to be introducing TestRig 2.0 into my environment. How do I know it is secure?
The TestRig team kept best security practices in mind during development. During normal operation the TestRig environment makes no changes to the host machine and no data is written to the host media. The environment, including the file system, runs entirely in memory. After booting the user is immediately placed into the network test control program. Standard methods of escaping an application (ctrl-c, ctrl-z, and ctrl-\) are disabled. However, it is possible that the control program may crash and put the user into an interactive shell.
All network communications between the TestRig environment and the central TestRig management server take place over SSL. No private or public keys are stored on the TestRig ISO. Test results are only sent to the defined host account - no results or information are shared with PSC or stored on the central TestRig management server. Aside from necessary network test applications there are no servers found on the TestRig ISO; ssh, http, smtp, etc are not installed on the ISO. Likewise, ssh, scp, ftp, telnet, netcat, and the like are not installed on the ISO. Lastly, apt and dpkg are also not found on the ISO. While perl is, by necessity, on the ISO no other compiled or interpreted languages are present.
12. Where are the results? How do I interpret them? Can you provide support for this?
After the completion of the tests the packaged results will be securely transferred to your designated host. The name structure of the results file is <UUID>-<N>.tgz where UUID is the unique universal identity string that corresponds to the ISO. You can match the UUID to the specific user and trouble ticket in the ISO list tab of your account on testrig.psc.edu. Untarring the package will create a ‘results’ directory. In that directory will be a number of files with that follow the <UUID>-<N>-<Test> pattern. For example:
F1F1BD16-C3D3-11E6-86F7-2B8E7D7DE884-1-hardware
F1F1BD16-C3D3-11E6-86F7-2B8E7D7DE884-1-tcpdump.gz
F1F1BD16-C3D3-11E6-86F7-2B8E7D7DE884-1-tcptrace
F1F1BD16-C3D3-11E6-86F7-2B8E7D7DE884-1-iperf
Each file contains the results of each test in ascii format. The exception to this is the tcpdump.tgz file which is in tcpdump format. Additionally, there will be a *-tcptrace file in the directory that contains a preliminary analysis of the tcpdump file using tcptrace.
Interpretation of the results is dependent on the experience of the network engineer. Providing detailed guides to interpreting test results is outside of the scope of this document. The TestRig team is generally not available to help with the diagnostic interpretation of these results but exceptions will be made on a case by case basis.
13. How do I pick the best perfSonar node?
Generally speaking the best perfSonar node to use is the one with the shortest path to your user. The goal is to reduce the number of hops and RTT to a minimum in order to get the best results. If your institution is hosting their own perfSonar node and the user is within that network then this would be the ideal node to use. However, when this isn't a option whatever node is closest will still provide useful results.
TestRig 2.0 will automatically suggest nodes based on the hostname or ip address of the host being tested. This is based on geolocation IP address lookups. This is not guaranteed to be accurate as it depends on the quality of the geoip database (in our case MaxMind). This also dones't guarantee that it is the closest node in terms of the network path. As an alternative we suggest using ESNet's perfSonar services directory at http://stats.es.net/ServicesDirectory/. You can enter the result into the custom perfSonar node section when you generate a new ISO.
14. Why can't my user connect to the perfSonar node I chose?
This generally happens when the user's host is not connected to a research and education network (R&E). A large number of perfSonar nodes are only connected to R&E networks like Internet2 and XSEDE. So a user who is making use of a commodity internet connection (such as through Comcast or Verizon) won't be able to contact those nodes. In these cases it's important that you pick a node that has commodity connectivty.
15. What if my question wasn't answered here?
Email support at testrig@psc.edu