GENOVA Suite Programs

The GENOVA Suite of computer programs for generalizability theory consists of GENOVA, urGENOVA, and mGENOVA. The three programs are coordinated with Brennan (2001), which provides an extensive treatment of generalizability theory. Brennan (2001) can be purchased from the publisher, Springer-Verlag (www.springer-ny.com). It is strongly recommended that users of these programs consult Brennan (2001) in order to interpret the output accurately.

There are complied versions of GENOVA, urGENOVA, and mGENOVA for both Macintosh PowerPCs and DOS/Windows-based PCs. Each of the programs in the GENOVA Suite may be distributed to others without obtaining the permission of the author(s). The folders for each program contain a "readme" file, a manual in pdf format, the application, and at least one sample input file.

Questions concerning these programs should be directed to:
Robert L. Brennnan
University of Iowa
CASMA, College of Education
210 Lindquist Center
Iowa City, IA 52242
(Tel: 319-335-5405)
(Email: robert-brennan@uiowa.edu)

  1. GENOVA for PC (16.9 MB)
    GENOVA is a ANSI FORTRAN computer program for univariate generalizability analyses with complete, balanced designs. It has both G study and D study capabilities. GENOVA was designed by R. L. Brennan and coded by J. E. Crick in the early 1980s. The file is large because the manual is a pdf version of a scanned file.
  2. urGENOVA for PC, (356 KB)
    urGENOVA is an ANSI C computer program for the estimation of variance components for unbalanced random effects G study designs. The program does not have D study capabilities. urGENOVA was designed and coded by R. L. Brennan.
  3. mGENOVA for PC, (660 KB)
    mGENOVA is an ANSI C computer program for multivariate generalizability analyses for a restricted class of designs. For these designs, the program has both G study and D study capabilities. mGENOVA was designed and coded by R. L. Brennan.
  4. Answers to Additional Exercises. Errata.
    Brennan's (2001) book on "Generalizability Theory" provides a set of exercises for each chapter, and Appendix I provides answers to about half of the exercises. Answers to the remaining exercises are provided here, as well as the errata for "Generalizability Theory".

Reference

Brennan, R. L. (2001). Generalizability theory. New York: Springer-Verlag.

Equating Recipes Open-source Code and Monograph

Equating Recipes provides a set of open-source functions written in ANSI C to perform all types of equating discussed by Kolen and Brennan (2004), as well as some other equating methods, including modified frequency estimation, kernel equating, and continuized log-linear equating. There are over 25,000 lines of code in Equating Recipes, including comments. The Equating Recipes code and the associated monograph/manual are freely available. The code is available under the open-source license conditions discussed in the ReadMe file distributed with the code and summarized in Section 1.5 of the monograph/manual. Older versions of some of the Equating Recipe functions are used in equating programs provided elsewhere on the CASMA website.

Equating Recipes is provided in a folder named “ER for distribution (date)”. This folder contains: (a) source files and header files; (b) main functions, data files, conversion tables, and output files for the examples in the monograph/manual; (c) the Equating Recipes monograph/manual and a ReadMe file; and (d) the Microsoft Visual Studio project (which is the only code distributed that is platform specific). All users should read Chapter 1 of the monograph/manual, as well as the ReadMe file.

  1. Equating Recipes for PC (2.33 MB)

Reference

Kolen, M. J., & Brennan, R. L. (2004). Test equating, scaling, and linking: Methods and practices (2nd ed.). New York: Springer-Verlag.

ESUM-CINEG Open-source VBA Excel Macros and Manual

In many, if not most testing programs that employ equating, the new form is equated to at least two previously equated forms, say k of them. In such cases, the final conversion table is typically taken as some weighted average of the k conversions, where each of the k conversions is based on some particular equating method. Typically, the results for multiple methods are examined for each link, a particular method is chosen for each link, and then the final conversion table is some weighted average of the one particular user-selected method for each link. ESUM-CINEG (Equating SUMmary) is intended to facilitate these tasks.

ESUM-CINEG is a set of Excel VBA (Visual Basic for Applications) macros. The user can think of ESUM-CINEG as a computer program that "sits on top'' of Excel, which means that after ESUM-CINEG is run, all the features and capabilities of Excel are available to the user to manipulate, compute, or plot any of the ESUM-CINEG results.

ESUM-CINEG does not perform the equating for any particular method; computer programs available on the CASMA website (www.education.uiowa.edu/casma) can be used to do so. (See, in particular, Equating Recipes by Brennan, Wang, Kim, & Seol, 2009, which is on the CASMA website.) Rather, ESUM-CINEG displays numerical and graphical results that can help a user select a method for each link and ultimately choose the final conversion for the new form.

ESUM-CINEG takes as input the conversion tables and related information for each of the equating methods employed with each of the k links. The output for ESUM-CINEG is of two types: (a) equating results and plots for each link (one worksheet per link); and (b) equating conversion tables and related plots for the "new'' form based on a weighted average (weights chosen by user) of the equating conversion tables for the user-selected method for each link. The user can obtain equating summaries for as many different sets of methods (one per link) and weights (one per link) as desired.

ESUM-RG Open-source VBA Excel Macros and Manual

ESUM-RG (Equating SUMmary for the Random Groups design) is an open source VBA enabled Excel workbook that reads in output produced by Equating Recipes (Brennan, Wang, Kim, & Seol, 2009), and then organizes the output into worksheets that facilitate selecting an equating method for operational use.

The main purpose of ESUM-RG is to facilitate selection of an appropriate equating method. ESUM-RG allows a user to view equating results from five different perspectives: conversion tables, moments and moment differences (relative to scores on Form Y), color (e.g., many-to-one conversions and gaps in rounded equated scale scores), relative frequency distributions of rounded equated scale scores; and a wide selection of graphs.

Any single ESUM-RG workbook is for scores on a single form of a test. For example, if there are three scores (say, Reading, Writing, and Math) for any form, and two forms are equated to a base or anchor form, then there should be six ESUM-RG workbooks. Note that, for ESUM-RG, the data collection design must be the random groups design, as discussed by Kolen and Brennan (2014).

References

Brennan, R. L., Wang, T., Kim, S., & Seol, J. (2009, September). Equating Recipes (Version 1) [Computer software and manual] (CASMA Monograph No. 1). Iowa City, IA: Center for Advanced Studies in Measurement and Assessment, The University of Iowa. (Available on http://www.education.uiowa.edu/casma)

Kolen, M. J., & Brennan, R. L. (2004). Test equating, scaling, and linking: Methods and practices (2nd ed.). New York: Springer-Verlag.

Equating/Linking Programs

The computer programs listed below can be used to conduct many of the equating analyses described in Kolen and Brennan (2004). Data sets from this book are included with some of the programs. Windows PC console and graphical user interface (GUI) versions and Macintosh OS9 console and OS10 GUI versions are available for at least some of the programs. The programs can be downloaded below.

  1. CIPE for PC Console, CIPE for PC GUI, CIPE for MAC OS9, CIPE for MAC OS10
    Conducts linear and equipercentile equating under the common-item nonequivalent groups design.

  2. RAGE-RGEQUATE for PC Console, RAGE-RGEQUATE for PC GUI, RAGE-RGEQUATE for MAC OS9, RAGE-RGEQUATE for MAC OS10
    Conducts linear equating and equipercentile equating under the random groups design using cubic spline and log-linear methods.

  3. Equating Error for PC Console, Equating Error for PC GUI, Equating Error for MAC OS9, Equating Error for MAC OS10
    Estimates bootstrap standard errors of linear equating and equipercentile equating under the random groups design.
  4. PIE for PC Console, PIE for PC GUI, PIE for MAC OS9, PIE for MAC OS10
    Conducts IRT true and observed scoring equating for dichotomously scored tests.
  5. POLYEQUATE for PC Console, POLYEQUATE for MAC OS9
    Conducts IRT true and observed scoring equating for dichotomously and polytomously scored tests.
  6. LEGS for PC Console, LEGS for PC GUI, LEGS for MAC OS10, LEGS for MAC Console
    Conducts linear and equipercentile linking.

Reference

Kolen, M. J. & Brennan, R. L. (2004). Test Equating, Scaling, and Linking: Methods and Practices (2nd ed.). New York: Springer-Verlag.

IRT Scale Transformation Programs

The computer programs listed below can be used to conduct various IRT scale transformations. Windows PC console and graphical user interface (GUI) versions and Macintosh OS9 console and OS10 GUI versions are available for at least some of the programs. The programs can be downloaded below.

  1. ST for PC Console, ST for PC GUI, ST for MAC OS9, ST for MAC OS10
    Conducts item response theory (IRT) scale transformations for dichotomously scored tests.
  2. POLYST for PC Console, POLYST for MAC OS9
    Conducts IRT scale transformations for dichotomously and polytomously scored tests.
  3. STUIRT for PC Console
    Conduct IRT scale transformations for mixed-format tests. 

Classification/Decision Consistency Programs

  1. BB-CLASS for PC, BB-CLASS for MAC (480 KB)
    BB-CLASS is an ANSI C computer program that uses the beta-binomial model (and its extensions) for estimating classification consistency and accuracy. BB-CLASS is intended to provide results for both the Hanson and Brennan (1990) and Livingston and Lewis (1995) procedures, although BB-CLASS has some capabilities that slightly extend these procedures. The BB-CLASS program, a manual, and the sample runs in the manual are provided.

    References

    Hanson, B. A., & Brennan, R. L. (1990). An investigation of classification consistency indexes estimated under alternative strong true score models. Journal of Educational Measurement, 27, 345-359.

    Livingston, S. A., & Lewis, C. (1995). Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32, 179-197.
  2. MULT-CLASS v3.0 for PC (202 KB)
    MULT-CLASS is a FORTRAN computer program that uses the compound multinomial model for estimating classification consistency and accuracy. MULT-CLASS is intended to provide results discussed in Lee (2005).

    References

    Lee, W. (2005, Revised 2008). Classification consistency and accuracy under the compound multinomial model (CASMA Research Report No. 13). Iowa City, IA: University of Iowa.
  3. IRT-CLASS v2.0 for PC (approximately 224 KB for PC)
    IRT-CLASS is a PC console version of a FORTRAN computer program that computes classification consistency and accuracy indices for raw and scale scores. IRT-CLASS is intended to be used for tests that are scaled using dichotomous, polytomous, or mixtures of different IRT models.

    Reference

    Lee, W. (2008). Classification consistency and accuracy for complex assessments using item response theory (CASMA Research Report No. 27). Iowa City, IA: University of Iowa. 
     
  4. NM-CLASS v1.0 for PC (approximately 240 KB for PC) NM-CLASS is an R program that computes classification consistency and accuracy indices using the normal approximation procedure discussed in Peng and Subkoviak (1980).

    References

    Kim, S. Y., & Lee, W. (2019). Classification consistency and accuracy for mixed-format tests. Applied Measurement in Education, 32, 97-115.

    Peng, C. J., & Subkoviak, M. J. (1980). A note on Huynh’s normal approximation procedure for estimating criterion-referenced reliability. Journal of Educational Measurement, 17, 359-368.

Other Programs

  1. POLYCSEM for PC Console, POLYCSEM for MAC OS9.
    POLYCSEM can be used to estimate conditional standard errors of measurement under an IRT model dichotomously and polytomously scored tests.

  2. COMPOSITE SCORE UTILITY Composite Score Utility R Code and Composite Score Utility Example, 
    Provides the R code for computing utility indexes for composite scores by Brennan (2015) and an example.

     

    Reference

    Brennan, R. L. (2015, May). Utility indexes for composite scores. (CASMA Research Report No. 42). Iowa City, IA: Center for Advanced Studies in Measurement and Assessment, The University of Iowa. (Available on the CASMA website).