Comparison  
     
Functional Testing Tools
 
 
Change the importance of criteria   Customize comparison information   Download comparison results to your computer   Create a copy of the current comparison  

View comparison overall rankingView comparison summary scoresView comparison detailed findingsView comparison graphsView comparison notes
 
 
 
  View detailed findings for all Criteria Groups. Please be patient while the data loads.  
View All
 
The current scoring system algorithm is exponential.

    Development  
    What tools are provided for developing code, content, and web pages.  
    Factor: Standard  
    Product Name Rational Performance Studio
1.5
SilkTest
5.0.2
WinRunner
6.0
e-Test Suite
4.2
 
    Manufacturer Rational Software Segue Mercury Interactive RSW Software, Inc.  
      Group Score:
142 out of 230
Total Score:
989 out of 1520
  Group Score:
138 out of 230
Total Score:
845 out of 1520
  Group Score:
170 out of 230
Total Score:
816 out of 1520
  Group Score:
136 out of 230
Total Score:
736 out of 1520
 
    Records Scripts  
    Weight: 3   Weighted Score:
16


  Weighted Score:
12


  Weighted Score:
12


  Weighted Score:
12


 
      Scripts for both GUI and Vuser are recorded using the Rational Robot tool. Three types of recording are possible: API, Proxy, and Network.   All applications are recorded using the SIlkTest application in the same process.   Scripts are generated with the WinRunner product. To record web applications, the Web Test add-in must be loaded. The recorder is simple to use. Scripts can be created and run quickly.   E-Tester is the tool used to record, modify, and run scripts.  
    Recording Process  
    Weight: 5   Weighted Score:
48


  Weighted Score:
48


  Weighted Score:
80


  Weighted Score:
48


 
      Scripts are recorded using Object-Oriented recording which makes a script more reusable because it does not rely on bitmap coordinates to locate an object.   SilkTest scripts are created using object-oriented recording. Scripts recorded using object oriented process are believed to be more reusable.   WinRunner allows for both Object-Oriented and analog (bitmap) recording. Giving it a lot more flexibility to test more types of applications such legacy RTE applications.   Scripts are created using object-oriented recording.  
    Scripting Languages Supported  
    Weight: 4   Weighted Score:
24


  Weighted Score:
16


  Weighted Score:
16


  Weighted Score:
24


 
      SQABasic is a Microsoft Basic-like language used for GUI scripts. VU is a C-like language that is used for Vuser scripts.   4Test is an object-oriented language used for SilkTest scripts.   Test Script Language is a proprietary language provided by Mercury.   VBScript. Scripts can also be extended using calls to external functions written in Visual Basic, C, C++, and Java.  
    Script Editor  
    Weight: 4   Weighted Score:
24


  Weighted Score:
32


  Weighted Score:
32


  Weighted Score:
24


 
      An editor for debugging, testing scripts and adding functionality is included.   SilkTest provides standard editor features such as debugging and color coding of keywords. A library browser is available from the help menu that allows the user to look-up methods associated with the objects in the script.   The editor has a feature called the Function Generator that makes it easier to add functions to the script by suggesting a function or by selecting one from a provided list. Also includes the standard editor features such as a debugger and breakpoints.   e-Test Suite Provides an editor to write scriptlets.  
    Native APIs Supported  
    Weight: 2   Weighted Score:
6


  Weighted Score:
6


  Weighted Score:
6


  Weighted Score:
4


 
      Windows APIs, .DLLs or custom-created functions can be called from the SQABasic language, just as they can in Microsoft Basic.   Segue's 4Test Language allows for call to native libraries.   Mercury's Test Script Language (TSL) provides native libraries to add more functionality to test scripts.   Scriptlets can be added to call native APIs as well as functions written in VB, C, or Java.  
    User Defined Functions  
    Weight: 4   Weighted Score:
24


  Weighted Score:
24


  Weighted Score:
24


  Weighted Score:
24


 
      Both the SQABasic and VU language allow for the creation of user-defined functions due to the fact that they are so closely related to Microsoft Basic and the C language.   Custom functions can be created to add functionality.   TSL allows for the creation of custom functions that can be called from within the script.   Functions can be written in the form of scriptlets which can be added to the script to add more specific functionality.  
 

    Application Types  
    The types of applications that can be tested (i.e., Web or Client-Server).  
    Factor: Standard  
    Product Name Rational Performance Studio
1.5
SilkTest
5.0.2
WinRunner
6.0
e-Test Suite
4.2
 
    Manufacturer Rational Software Segue Mercury Interactive RSW Software, Inc.  
      Group Score:
64 out of 80
Total Score:
989 out of 1520
  Group Score:
48 out of 80
Total Score:
845 out of 1520
  Group Score:
80 out of 80
Total Score:
816 out of 1520
  Group Score:
32 out of 80
Total Score:
736 out of 1520
 
    Applications Supported  
    Weight: 5   Weighted Score:
64


  Weighted Score:
48


  Weighted Score:
80


  Weighted Score:
32


 
      Windows, Client Server, HTTP, Jolt, Oracle, Socket, ODBC, Sybase, Tuxedo, SAP, and CORBA   Windows, Client/Server, Web, Database, Java   Windows, Client/Server, Java, Web, RTE, ERP, CRM   Web Only  
 

    General  
    Basic information about the product.  
    Factor: Standard  
    Product Name Rational Performance Studio
1.5
SilkTest
5.0.2
WinRunner
6.0
e-Test Suite
4.2
 
    Manufacturer Rational Software Segue Mercury Interactive RSW Software, Inc.  
      Group Score:
150 out of 215
Total Score:
989 out of 1520
  Group Score:
91 out of 215
Total Score:
845 out of 1520
  Group Score:
141 out of 215
Total Score:
816 out of 1520
  Group Score:
111 out of 215
Total Score:
736 out of 1520
 
    Display OS  
    Weight: 1   Weighted Score:
2


  Weighted Score:
3


  Weighted Score:
2


  Weighted Score:
2


 
      WIN   WIN - UNIX   WIN   WIN  
    Documentation  
    Weight: 4   Weighted Score:
32


  Weighted Score:
16


  Weighted Score:
24


  Weighted Score:
24


 
      Documentation is provided in both hard copy and .pdf format. The numerous manuals provided cover all aspects of the testing process and explains in great detail the process followed by Rational.   Documentation includes tutorials for web and windows applications. The documentation, the analyst believes fails to demonstrate the full functionality of SilkTest. Training would definitely be recommended.   Documentation is available in .pdf and hard copy; includes a useful tutorial that does a good job of pointing out the useful features of WinRunner.   Each tool comes with its own user manual. Getting started manual was helpful in learning the process RSW follows to test software. Available in .pdf format.  
    Ease of Use  
    Weight: 3   Weighted Score:
16


  Weighted Score:
12


  Weighted Score:
16


  Weighted Score:
16


 
      When Rational PerformanceStudio is first installed, the number of tools provided can seem overwhelming. But once the user begins working with the Rational Robot tool, they should find an intuitive tool that allows them to accomplish their tasks with relative ease. The tutorials provided give the user a good understanding of how the pieces fit together.   SilkTest is very structured in the way in adheres to a strict testing paradigm. It makes it easy to follow a testing process of creating test cases that only test on function at a time and grouping in to a single test plan.   Recording and editing scripts are easily accomplished with the WinRunner product. The interface is navigated easily. After the completion of the provided tutorial the user should be able to record, edit, and run a script in no time.   With e-Test Suite, there is a lot to learn, especially the script recorder, but the tutorials provided do a good job of familiarizing the user with the product's functionality and features. I was able to record a script and test within a few minutes.  
    Additional Software Required  
    Weight: 1   Weighted Score:
5


  Weighted Score:
3


  Weighted Score:
2


  Weighted Score:
1


 
      All needed software is supplied by the Rational PerformanceStudio.   SilkRadar is needed to provide bug tracking functionality.   Test Suite is needed for Bug Tracking and Test Plan integration.   Third party error tracking, data generation and source code control software if desired.  
    General Test Paradigm  
    Weight: 1   Weighted Score:
3


  Weighted Score:
3


  Weighted Score:
3


  Weighted Score:
4


 
      A repository is created to store the assets for the test. A script is then recorded using the Robot tool. The script can then be customized to include custom functions. It is then played back in the Robot tool. After test completion, results are then viewed in the Log viewer at which time the failures or bugs can then be entered into the ClearQuest product. Reports can then be made into reports and analyzed with the Test Manager product.   A test plan is created either by hand or by using the provided wizard which includes adding a test frame with the application to be tested. A base state is the recorded so SilkTest can reset the application after each test execution. Scripts are then recorded and added to the plan where the can be modified to add functionality. The tests are ran and the results analyzed.   Scripts are created by first running the Rapid Test Script wizard so the product can learn all the objects associated with the application or by simply recording a business process and expected results. Scripts can be edited to add functionality to test certain features or criteria. Run the test and then analyze the results with the Results viewer.   The user chooses or creates a workspace to place recorded scripts. Scripts are then recorded and edited within e-Tester. Very simple and straight forward.  
    Platform Tested  
    Weight: 1   Weighted Score:
0


  Weighted Score:
0


  Weighted Score:
0


  Weighted Score:
0


 
      WIN NT 4.0 SP6 Pentium Pro II 350 MHz 256K Ram   WIN NT 4.0 SP6 Pentium Pro II 350 MHz 256K Ram   WIN NT 4.0 SP6 Pentium Pro II 350 MHz 256K Ram   WIN NT 4.0 SP6 Pentium Pro II 350 MHz 256K Ram  
    Product Positioning  
    Weight: 1   Weighted Score:
0


  Weighted Score:
0


  Weighted Score:
0


  Weighted Score:
0


 
      Rational Software provides the industry's only team unifying solutions, encompassing market-leading tools for all team members, integrated best practices arising from more than 18 years of software engineering leadership, and comprehensive professional services.   SilkTest from Segue Software is the industry's leading regression testing product for e-business applications. Whether you're testing a Web, Java or traditional client/server application, SilkTest supports your organization's needs.   WinRunner is an integrated, functional testing tool for your entire enterprise. By capturing, verifying, and replaying user interactions automatically, WinRunner identifies defects and ensures that business processes, which span across multiple applications and databases, work flawlessly the first time and remain reliable.   e-Tester is the only functional test tool that doesn't force you to become an expert programmer in order to write comprehensive tests.  
    System Requirements  
    Weight: 2   Weighted Score:
4


  Weighted Score:
6


  Weighted Score:
6


  Weighted Score:
8


 
      Win NT or 2000, 500MB free disk space, 64MB RAM. Pentium 133 or better.   Windows 9x, NT 100MB disk space 64MB RAM.   Win 95/NT 64MB RAM 100MB Disk Space.   Win NT 4.0 or Windows 9x, Windows 2000, 64MB RAM, 70MB disk space, Pentium Pro or better.  
    Technical Support  
    Weight: 4   Weighted Score:
24


  Weighted Score:
16


  Weighted Score:
24


  Weighted Score:
24


 
      Technical support is available by phone (9:00AM - 6:00PM) or email. International Support is also available for Asia Pacific, Europe, Middle East, and Africa.   Phone support is available worldwide along with email and and web support forms.   Mercury Interactive has many different support programs available. They are available by phone throughout the world with varying hours.   Product support available by phone between 9AM and 6PM EST. Support questions can also be submitted through the company website. The staff was knowledgeable, and returned calls in a timely fashion.  
    Training Available  
    Weight: 5   Weighted Score:
64


  Weighted Score:
32


  Weighted Score:
64


  Weighted Score:
32


 
      Rational University provides training throughout the United States and Canada. Classes not only focus on products but also the principles of functional testing.   Classes are available across the United Stated. Consulting services are also available   Mercury provides Partner-Certification programs, On-Site Training, WorldWide Training along with offering its own class at different sites throughout the U.S.   Training is available onsite; consulting and partner certification is also available.  
 

    Reporting  
    Environment for retrieving and manipulating stored reports.  
    Factor: Standard  
    Product Name Rational Performance Studio
1.5
SilkTest
5.0.2
WinRunner
6.0
e-Test Suite
4.2
 
    Manufacturer Rational Software Segue Mercury Interactive RSW Software, Inc.  
      Group Score:
88 out of 140
Total Score:
989 out of 1520
  Group Score:
84 out of 140
Total Score:
845 out of 1520
  Group Score:
60 out of 140
Total Score:
816 out of 1520
  Group Score:
48 out of 140
Total Score:
736 out of 1520
 
    Log Viewer  
    Weight: 4   Weighted Score:
32


  Weighted Score:
24


  Weighted Score:
24


  Weighted Score:
16


 
      The log viewer can be set to display automatically after test completion. Shows verification points included in the script as well as Pass or Fail, time and date of test.   The results file automatically displays after the completion of the test cases. The results appear in a window within the SilkTest application not as a separate application.   All events can be viewed in the Test Results Viewer which can be configured to display automatically upon test completion.   The log viewer is simple with few features. It gives the user the ability to export the log to a text file.  
    Generates Reports  
    Weight: 4   Weighted Score:
24


  Weighted Score:
24


  Weighted Score:
8


  Weighted Score:
0


 
      Summary reports can be created using the Test Manager product that's included with the product suite. The summary report includes Project name, script name, and type and description. Defect reports can be generated from the ClearQuest as long as the defects were entered from the log viewer.   Completion reports are generated to track the progress of a test plan. After a test case is ran a report is automatically generated that displays the results of the ran tests. Information includes pass/fail, started and elapsed time.   Results can be generated into a text-based report. Further reporting requires the use of the Test Suite.   No reports are generated for results. All test information is written to log files. Third party tools can be used.  
    Log Filters  
    Weight: 3   Weighted Score:
16


  Weighted Score:
12


  Weighted Score:
12


  Weighted Score:
8


 
      Filters can also be created using the Log Viewer. Included is the ability to include event types.   Filters can be set to provide as much or little information as the user wants. Results files may also be merged with other results files.   The type of events to be shown in the results logs can be selected from the Test Results viewer.   Log options allow the user to report failures only or all details. No other filtering is available.  
    Runtime Message Logging  
    Weight: 4   Weighted Score:
16


  Weighted Score:
24


  Weighted Score:
16


  Weighted Score:
24


 
      All messages are sent to log files.   Messages are written automatically to the log file. Scripts can also be modified to pass any information to the log file the user may find useful.   All events are written directly to the log files.   During playback, if the script all messages can be viewed as they occur, they are also written to the log file.  
 

    Test Controller Criteria  
    The feature of the overall control criteria for test execution.  
    Factor: Standard  
    Product Name Rational Performance Studio
1.5
SilkTest
5.0.2
WinRunner
6.0
e-Test Suite
4.2
 
    Manufacturer Rational Software Segue Mercury Interactive RSW Software, Inc.  
      Group Score:
97 out of 175
Total Score:
989 out of 1520
  Group Score:
44 out of 175
Total Score:
845 out of 1520
  Group Score:
73 out of 175
Total Score:
816 out of 1520
  Group Score:
109 out of 175
Total Score:
736 out of 1520
 
    Runtime Test Control  
    Weight: 3   Weighted Score:
16


  Weighted Score:
8


  Weighted Score:
12


  Weighted Score:
16


 
      Many parameter options are available to control the playback of the script. Some include command execution delay, error recovery, logs, and diagnostic tools to be used.   Runtime options available to control tests include Type of Network, agent name, browser type and few others.   Some of the options that can be set include checkpoint timeouts, some error conditions, and speed of execution.   Options that can be set to control playback include, but are not limited to, error recovery, page timeout, and use of DataBanks.  
    Supports Forms  
    Weight: 2   Weighted Score:
8


  Weighted Score:
6


  Weighted Score:
6


  Weighted Score:
6


 
      Yes, datapools can be used to generate information for a more realistic test of form input.   SilkTest provides support for forms.   WinRunner provides support for forms.   e-Test Suite provides support for forms.  
    Parameterization  
    Weight: 1   Weighted Score:
5


  Weighted Score:
2


  Weighted Score:
3


  Weighted Score:
3


 
      User parameters such as login or registration information can be either imported from a file or generated by Rational through the use of Datapools.   Parameterization is accomplished by adding statements to the script that identify it as a data driven test and then specifying the data type and data file to use.   Tests can be parameterized by importing data from a file, a database, or by using the DataDriver Wizard.   DataBanks can be used to provide user parameters to make a more realistic test.  
    Import Results Files  
    Weight: 5   Weighted Score:
48


  Weighted Score:
0


  Weighted Score:
0


  Weighted Score:
64


 
      Files can be imported to be used as verification points called file comparison. This test can check that a file created within the application contains the correct information.   SilkTest does not allow the user to import results files.   Documentation mentions expected results files but does not mention any way to import them.   Using Databanks, the user can import not only results but the input data also.  
    Database Verification  
    Weight: 4   Weighted Score:
8


  Weighted Score:
16


  Weighted Score:
40


  Weighted Score:
8


 
      No specific functionality is available to facilitate this feature. A script could be written to achieve this functionality.   No specific feature is supplied to test that a transaction was successfully written to a database. It is possible to write a query within the script to test this though.   Database checkpoints can be added to the script to test that a transaction was successfully entered into a database.   No specific feature is available to facilitate this feature. Can be accomplished by writing a scriptlet.  
    Supports Proxy Server Connections  
    Weight: 3   Weighted Score:
12


  Weighted Score:
12


  Weighted Score:
12


  Weighted Score:
12


 
      Yes   Yes   Yes.   Yes  
 

    Tools and Options  
    The regular utilities and configurable aspects of the product.  
    Factor: Standard  
    Product Name Rational Performance Studio
1.5
SilkTest
5.0.2
WinRunner
6.0
e-Test Suite
4.2
 
    Manufacturer Rational Software Segue Mercury Interactive RSW Software, Inc.  
      Group Score:
388 out of 580
Total Score:
989 out of 1520
  Group Score:
380 out of 580
Total Score:
845 out of 1520
  Group Score:
232 out of 580
Total Score:
816 out of 1520
  Group Score:
240 out of 580
Total Score:
736 out of 1520
 
    Browser Selection  
    Weight: 3   Weighted Score:
12


  Weighted Score:
12


  Weighted Score:
0


  Weighted Score:
8


 
      The browser type can be selected from the GUI playback menu.   SilkTest allows the user to select a browser.   Browser selection is not available.   e-Test Suite provides the ability to generate a Netscape 4.0 or higher scripts. Scripts created are compatible for Microsoft Internet Explorer by default.  
    Bug Tracking  
    Weight: 5   Weighted Score:
64


  Weighted Score:
48


  Weighted Score:
32


  Weighted Score:
0


 
      Errors or defects can be entered automatically from the log viewer into the ClearQuest product.   SilkTest integrates directly with SilkRadar to accommodate bug tracking functionality. SilkRadar is purchased separately from SilkTest.   Test Director is needed to track defects.   This functionality is not supported, but third-party tools can be used if desired.  
    Data Driven Tests  
    Weight: 5   Weighted Score:
64


  Weighted Score:
48


  Weighted Score:
48


  Weighted Score:
64


 
      Data driven tests are supported, and files supplying test and results data can be imported easily.   Data driven tests are supported by SilkTest.   The Data Driver Wizard can be used to parameterize the scripts. Microsoft Query or Data Junction can also be used to import data.   DataBanks can be used to map external data sources to facilitate data-driven tests.  
    Error Recovery Feature  
    Weight: 5   Weighted Score:
48


  Weighted Score:
80


  Weighted Score:
48


  Weighted Score:
64


 
      Error recovery option can be set for Script command failures or verification point failure, also the user can set recovery for unexpected windows that appear.   SilkTest error recovery feature returns the application to a base-state before each test in a test plan is executed. Which makes running multiple tests much easier and accurate.   Exceptions are created under the tools menu. They direct WinRunner on how to handle execution when an error occurs. A number of exceptions for web-based applications are provided.   Error recovery options are set under the Options menu in e-Tester. Provides a table that allows the user to select what actions to take when an error occurs.  
    Runs Unattended  
    Weight: 4   Weighted Score:
0


  Weighted Score:
32


  Weighted Score:
8


  Weighted Score:
24


 
      This functionality is not available for functional testing.   Though there is no scheduler to automatically run tests at a specified time. The error recovery feature allows tests to be started at the end of a day and ran over night by resetting the application to a base state after each execution.   This functionality is available using the Test Director.   This functionality is provided with e-Monitor  
    Source Code Control Systems  
    Weight: 5   Weighted Score:
32


  Weighted Score:
32


  Weighted Score:
16


  Weighted Score:
16


 
      Rational Perfmance Studio provides some useful functinoality, but is not directly integrated with a source code control system.   SilkTest integrates with PVCS.   This functionality is not provided, but a third-party tool could be used if desired.   This functionality is not provided, but a third-party tool could be used if desired.  
    Test Plan Integration  
    Weight: 5   Weighted Score:
64


  Weighted Score:
80


  Weighted Score:
48


  Weighted Score:
16


 
      TestManager tool helps the user plan and synchronize tests. Included with the PerformanceStudio 1.5   This functionality is built in to SilkTest and a key feature in its architecture.   WinRunner can integrate directly with the Test Director which is Mercury's Test Plan Management system.   This functionality is not included but the product is designed to integrate with third party tools.  
    Link Checker  
    Weight: 5   Weighted Score:
64


  Weighted Score:
48


  Weighted Score:
32


  Weighted Score:
48


 
      Rational SiteChecker is included with the PerformanceStudio; it is a comprehensive tool that can check the integrity of your website links internally and externally.   This functionality is included with a new feature called LinkTester   To check if links are valid, a checkpoint must be inserted into the script. They can test individual links or all links within a frame.   Resource validation can run automatically after the completion of the test. Results can be exported to a text file.  
    Generation of Test Data  
    Weight: 4   Weighted Score:
40


  Weighted Score:
0


  Weighted Score:
0


  Weighted Score:
0


 
      Datapools are used to create user data to simulate a more realistic test scenario.   The product does not support the generation of test data.   Data can be mapped using the Data Driver Wizard but not created.   The product does not support the generation of test data, but third party tools can be used if desired.  
 

    Security  
    Security criteria may include, but is not limited to, standards such as SSL and SET, protection of administrative interfaces, user and administrator access, key management (PKI), encryption, authentication, firewalls, virus protection, and other various security tools and features.  
    Factor: Standard  
    Product Name Rational Performance Studio
1.5
SilkTest
5.0.2
WinRunner
6.0
e-Test Suite
4.2
 
    Manufacturer Rational Software Segue Mercury Interactive RSW Software, Inc.  
      Group Score:
60 out of 100
Total Score:
989 out of 1520
  Group Score:
60 out of 100
Total Score:
845 out of 1520
  Group Score:
60 out of 100
Total Score:
816 out of 1520
  Group Score:
60 out of 100
Total Score:
736 out of 1520
 
    Supports SSL and SHTTP  
    Weight: 5   Weighted Score:
48


  Weighted Score:
48


  Weighted Score:
48


  Weighted Score:
48


 
      The product supports SSL and SHTTP.   The product supports SSL and SHTTP.   The product supports SSL and SHTTP.   The product supports SSL and SHTTP.  
    Supports Username and Password Login  
    Weight: 3   Weighted Score:
12


  Weighted Score:
12


  Weighted Score:
12


  Weighted Score:
12


 
      The product supports username and password login and will automatically accept and process user verification requests.   The product supports username and password login and will automatically accept and process user verification requests.   The product supports username and password login and will automatically accept and process user verification requests.   The product supports username and password login and will automatically accept and process user verification requests.  
 

  2000 Intraware, Inc.