ERIC Identifier: ED391989
Publication Date: 1995-00-00
Author: Kapes, Jerome T.
Source: ERIC Clearinghouse on Counseling and Student Services Greensboro NC.
Locating and Evaluating Career Assessment Instruments. ERIC Digest.
OVERVIEW
For the purpose of both locating and evaluating career assessment instruments, there are three primary sources. Best known among these are the Buros Institute's publication "Tests in Print" and its comparison set of reviews in the "Mental Measurements Yearbooks (MMY)." A second source, which first became available in 1983-84, is "Tests and Test Critiques (TC)." Both include a listing and brief description of most tests commercially available in English speaking countries (i.e., "Tests in Print" and "Tests") as well as periodically published volumes of test reviews (i.e., "MMY" & "TC")
The third source, which is published by the National Career Development Association, is "A Counselor's Guide to Career Assessment Instruments." This book, first published in 1982 and every six years since, contains reviews of the most prominent career assessment instruments as well as brief descriptions of most others commercially available. In addition, this book also includes chapters on selecting, evaluating, using, and interpreting career relevant tests.
There are a number of other sources that focus on specialized aspects of career assessment. Also, certain journals publish test reviews or articles that provide evidence of the quality of specific career assessment instruments. These qualities typically are evaluated under the categories of norms, reliability and validity. The American Educational Research Association (AERA), American Psychological Association (APA) and National Council on Measurement in Education (NCME) jointly publish "Standards for Educational and Psychological Testing," which provides guidance for both test publishers and users of all types of tests (AERA, APA & NCME, 1985). In addition, the American Counseling Association (formerly the American Association for Counseling and Development, 1989) has produced its own guidelines, "Responsibilities of Users of Standardized Tests," which provides additional help for users in counseling situations.
Much information is available to help locate and evaluate career assessment instruments, but it is the users who must employ these resources to make their own judgments about the appropriateness of a particular instrument for a specific situation. The purpose of this digest is to help users locate and organize information that will improve their evaluations.
LOCATING INSTRUMENTS
Although "Tests in Print IV" (1994) and "Tests" (1991) are the most comprehensive listings of all tests, those wishing to locate a career assessment instrument may find it more useful to consult "A Counselor's Guide to Career Assessment Instruments" - 3rd Edition (Kapes, Mastie and Whitfield, 1994). This recent edition contains reviews of 52 prominent instruments along with an Additional Instruments chapter which briefly describes an additional 245 instruments. The entire 297 instruments are listed alphabetically in a User's Matrix that categorizes each entry by Characteristics (achievement, aptitude, interest, values/satisfaction/environments, career development/maturity, personality) and Use level (elementary, junior high/middle school, senior high, 2- or 4-year college, adult education/ training, business and industry/employment, disabled or disadvantaged). Those interested in locating a test for a specific purpose can use this matrix to identify instruments that may be appropriate.If the instruments selected in the initial search are among the 52 with a complete review, the user can consult the reviews to further narrow the choice. Each entry includes a section of publisher-provided information that includes target population, statement of purpose, titles of subtests, scales and scores, forms and levels, date of most recent edition, languages available, time, norm groups, results reported, format, scoring, computer software, costs, comments, and published reviews. The review section is divided into the following headings: Description, Use in Counseling, Technical Considerations, Computer-Based Version (if available), Overall Critique, and References. If the instruments on which further information is needed are not among the 52 reviewed, the Additional Instruments chapter can be consulted to obtain the publisher, date and intended population on any of the 245 additional instruments. In addition, citations for all reviews published in the "Mental Measurements Yearbooks," "Test Critiques," previous editions of "A Counselor's Guide" as well as several other sources are listed. A brief description for each test is also included.
For those instruments not reviewed in "A Counselor's Guide" (3rd edition) or for a second opinion, the reader can consult one or more reviews cited in either the Additional Instruments chapter or at the end of the publisher information. If a "Mental Measurements Yearbook (MMY)" is to be consulted, it is necessary to know both the volume (Sixth, 1965 through Eleventh, 1992) and test number. The "MMY" entries typically contain brief publisher information along with two independent reviews. The reviews themselves are not divided into sections and tend to focus primarily on psychometric characteristics. Additional references are also provided.
To access "Test Critiques (TC)," it is also necessary to know the volume (Volume 1, 1984 through Volume 10, 1994). Each of these entries are written by a single reviewer and are divided into five categories (Introduction, Practical Applications/Use, Technical Aspects, Critique, and References). In addition to "MMY" and "TC" it may also be useful to consult the first (1982) and second (1988) editions of "A Counselor's Guide" or any of the other sources listed in the chapter on Sources of Information about Testing and Career Assessment in the third edition, which is an annotated bibliography of sources.
EVALUATING INSTRUMENTS
Many sources exist that could aid a user to evaluate the potential usefulness of a career assessment instrument. The previously mentioned AERA, APA and NCME (1985) document provides guidance for both test publishers and users in the form of essential, conditional and secondary standards. The standards, for example, call for a technical manual to be made available by the publisher so that any user can obtain information about the norms, reliability and validity of the instrument as well as other relevant topics. It should be pointed out that, although the publisher typically provides evidence of this type from studies conducted with subjects for whom the instrument is intended, it may be necessary for the user to obtain data from other sources that better reflect the use intended for a particular application. This can be done from studies published in the literature (e.g., in the "Measurement and Evaluation in Counseling and Development" journal or the "Career Development Quarterly") or from studies conducted on data collected by the user.In addition to the technical qualities of norms, reliability and validity, there are many other qualities of a career assessment instrument that need to be evaluated before selection for a particular use. In his chapter in "A Counselor's Guide" on "Selecting a Career Assessment Instrument," Mehrens covers many of these, including types of scores and interpretation materials, appropriateness for various groups and a host of practical issues (e.g., qualifications of users, time, costs, and publisher support). In addition, he provides a "Test Evaluation Outline" that is included here to assist the user to systematically identify and collect all information necessary to conduct an adequate evaluation (Mehrens, 1994, p. 29):
1. State
your purpose for testing
2. Describe the group that will be
tested (for example, age or grade)
3. Name of test
4.
Author(s)
5. Publisher
6. Copyright date(s)
7. Purpose and
recommended use as stated in the manual
8. Grade/age levels for
which the instrument was constructed
9. Forms: Are equivalent
forms available? What evidence is presented on the equivalence of
the forms?
10. Format: comment on legibility, attractiveness, and
convenience
11. Cost
12. Content of test and types of items
used
13. Administration and timing requirements
14. Scoring
processes available (e.g., machine scoring)
15. Types of derived
scores available
16. Types and quality of norms
17. Adequacy of
reliability evidence presented in the manual
18. Validity
evidence
19. General quality of administrative, interpretative and
technical manuals
20. Comments about the instrument by outside
reviewers
SUMMARY
There are many sources to aide in locating and evaluating career assessment instruments. The primary sources are "A Counselor's Guide to Career Assessment Instruments," the "Mental Measurement Yearbooks," and "Test Critiques." The AERA, APA and NCME Standards provide guidance to publishers and users on the qualities of norms, reliability, and validity as well as many other considerations that affect test use. However, the bottom line is that the user is responsible to make the final judgment about the appropriateness of a particular instrument for a specific use.REFERENCES
American Association for Counseling and Development (1989). Responsibilities of users of standardized tests (AACD/AMECD Policy Statement). Alexandria, VA: Author.American Educational Research Association, American Psychological Association, National Council on Measurement in Education. (1985). Standards for educational and psychological testing. Washington, D.C.: American Psychological Association.
Kapes, J.T., Mastie, M.M., & Whitfield, E.A. (Eds.). (1994). A counselor's guide to career assessment instruments (3rd edition). Alexandria, VA: National Career Development Association.
Keyser, J.D., & Sweetland, R.C. (1984-94). Test critiques (Vols. 1-10). Austin, TX: PRO-ED. Kramer, J.J., & Conoley, J.C. (Eds.). (1992). Eleventh mental measurements yearbook. Lincoln, NE: Buros Institute of Mental Measurement.
Mehrens, W.A. (1994). Selecting a career assessment instrument. In J.T. Kapes, M.M. Mastie, & E.A. Whitfield (Eds.), A counselor's guide to career assessment instruments (3rd edition). Alexandria, VA: National Career Development Association.
Murphy, L.L., Conoley, J.C., Impara, J.C., & Mitchell, J.V. (Eds.). (1994). Tests in print IV. Lincoln, NE: Buros Institute of Mental Measurement.
Sweetland, R.C., & Keyser, D.J. (Eds.). (1991). Tests: A comprehensive reference for assessments in psychology, education, and business (3rd edition). Austin, TX: PRO-ED.
Jerome T. Kapes, Ph.D., NCC, NCCC, is a professor of Educational Psychology and a faculty member in the Counseling Psychology, Educational Psychology Foundations and Vocational Education programs at Texas A&M University in College Station, Texas.
ERIC Digests are in the public domain and may be freely reproduced and disseminated. This publication was funded by the U.S. Department of Education, Office of Educational Research and Improvement, Contract No. RR93002004. Opinions expressed in this report do not necessarily reflect the positions of the U.S. Department of Education, OERI, or ERIC/CASS.
Title: Locating and Evaluating Career Assessment Instruments. ERIC Digest.
Document Type: Information Analyses---ERIC Information Analysis Products (IAPs) (071); Information Analyses---ERIC Digests (Selected) in Full Text (073);
Descriptors: Aptitude Tests, Career Choice, Career Counseling, Career Exploration, Career Guidance, Career Planning, Careers, Evaluation Methods, Evaluation Research, Interest Inventories, Occupational Tests, Test Interpretation, Test Reviews, Test Selection, Testing
Identifiers: ERIC Digests