INTERNATIONAL RESEARCH GROUP MEASURING DEVELOPMENT

AERA 1997, CHICAGO
Symposium

Modeling Development: Solving Methodological Problems with Rasch Analysis

Organizer, Theo L. Dawson, Graduate School of Education, University of California at Berkeley

Chair, Trevor Bond, James Cook University, Townsville, Australia

Discussant, Mark Wilson, Graduate School of Education, University of California at Berkeley

• Identifying Learner Characteristics Relevant to Children's Understanding of Area Concepts
  Trevor Bond & Kellie Parkinson, School of Education James Cook University, Townsville, Australia
  
  
• Reasoning About "Good Education": A Developmental Analysis
  Theo Linda Dawson, Graduate School of Education, University of California at Berkeley
  
  
• A Rasch Analysis of a Set of Piagetian-Based Written Problems Representing Different Forms of Thought and Different Logical Operations
  William Gray, Educational Psychology, University of Toledo, Ohio
  
  
• Beyond Rasch in the Measurement of Stage-like Development
  Karen Draney, Graduate School of Education, University of California at Berkeley

Abstracts

Identifying Learner Characteristics Relevant to Children's Understanding of Area Concepts

A representative sample of forty-two students (age from 5 years; 3 months to 13 years; 1 month.) were randomly selected from a total student population of 142 students at a small private primary school in northern Australia. Those children's understandings of area concepts taught during the primary school years were assessed by their performance in two testing situations: The first consisted of a written test of ability to solve 'area' problems with items drawn from school texts, school examinations and other relevant curriculum documents. The second was an individual interview for each child where four 'area' tasks such as the 'Meadows and Farmhouse Experiment' taken from Chapter 11 of The Child's Conception of Geometry (Piaget, Inhelder and Szeminska, 1960, pp. 261 -301) were administered.

Rasch analysis based on the Partial Credit Model provided a finely detailed quantitative description of the developmental and learning progressions revealed in the ensuing data. It is evident that the mathematics curriculum does not satisfactorily match the learner's developmental sequence at some key points. Moreover, the children's ability to conserve area on the Piagetian tasks, rather than other learner characteristics, such as age and school grade seems to be a prerequisite for complete success on the mathematical test of area. The discussion focuses on the assessment of developmental (and other) characteristics of school-aged learners and suggests how curriculum and school organization might better capitalize on such information in the design and sequencing of learning experiences.

Reasoning About "Good Education": A Developmental Analysis

To assess the development of reasoning about the nature of a "good education," 116 semi-structured clinical interviews were conducted with a largely middle- and upper- middle-class group of participants ranging in age from 5 to 85. Most of the participants took part in Armon's (1984) life-span longitudinal study of moral reasoning and reasoning about the good, but were previously unscored. All interviews were conducted as specified by Armon (1984). The stage of each scorable response was determined using Common's General Stage Model (Commons, 1984), which identifies (in developmental sequence) primary, concrete, abstract, formal, systematic, and metasystematic stages of performance. Using criteria from the General Stage Scoring Manual, all scorable statements from each of the interviews were identified, randomly ordered, and scored blindly.

The scores were submitted to a Rasch analysis, the results of which (1) support the sequence of development proposed in the General Stage Model, (2) indicate that individuals reasoned relatively uniformly throughout their interviews, and (3) provide evidence of 'gappiness' that supports the concept of a step-like progression from stage to stage.

The Good Education scores of participants from Armon's study (N=77) were then added to the data from a previous Rasch analysis of her results. In the earlier analysis, it was found that Kohlberg's Moral Reasoning measure and Armon's Good Life, Good Work, Good Friendship, and Good Person measures appeared to be tapping the same underlying dimension of reasoning. Reanalysis of the data, including the Good Education scores, reveals that the General Stage Scoring System also appears to tap this single dimension. The theoretical and practical implications of this finding will be discussed.

References

Armon, C. (1984b). Ideals of the good life: Evaluative reasoning in children and adults. Unpublished Doctoral dissertation, Harvard, Boston.

Commons, M. L., Richards, F. A., with Ruf, F. J., Armstrong-Roche, M., & Bretzius, S. (1984). A general model of stage theory. In M. Commons, F. A. Richards, & C. Armon (Eds.), Beyond Formal Operations (pp. 120-140). New York: Praeger.

A Rasch Analysis of a Set of Piagetian-Based Written Problems Representing Different Forms of Thought
and Different Logical Operations

This is an analysis of a data set of 553 participants who answered 24 items from How Is Your Logic?. Mean age was 16.66 (SD = 4.97), ranging from 10.95 to 48.46. Males were 55.3% of the sample. Within a two week period, participants solved 24 written problems that represent different formal operations and different concrete operations. Each problem required a constructed response and answers were classified according to the form of thought required for a correct answer and the form of thought produced by participants on the problems.

Rasch analysis provided support for the construction of the problems as congruent with the meaning of the underlying dimension of operational thought. Problem placements clearly separated the concrete problems and the formal problems with no concrete problem being more difficult than a formal problem and no formal problem being easier than a concrete problem.

Within the concrete problems, the seriation problems were the easiest and the multiplication of classes problems were more difficult. Within the formal problems, the combination problems were easier than, and separated from, the remaining formal problems. Making correct inclusions, proportional reasoning, and one permutation problem were the next easiest, although that permutation problem was not in its expected position. Denying incorrect inclusions problems were the most difficult along with the proportional reasoning explanation problems. All explanation problems were more difficult that their corresponding judgment problems. These results support recent suggestions that Rasch analysis is appropriate when analyzing data generated from a good developmental theory.

Beyond Rasch in the Measurement of Stage-like Development

One of the assumptions inherent in a Rasch model approach to development is that the patterns of the item difficulty estimates are sufficient to validly describe the student responses. This is the usual assumption in standardized tests, where the growth of students from lower achievement to higher achievement is usually conceived of as "smooth". This is distinctly not the case with stage-like development in general, and especially not so in the case of Piagetian and Neo-Piagetian theories, where the achievement of new equilibria at successive stages would seem to imply that the students' understandings of items would change from stage to stage, and hence, that the item parameters may well be re-arranged as the students move from stage to stage. There is a general class of item response models, described by Mislevy and Verhelst (1990), designed to model the situation where different subjects employ different solution strategies: we assume that the strategies are not observed, but that the differential relationship between the strategies and certain classes of items is known a priori. This general approach has been applied to the case of stage-like development by a number of authors under the title of the "Saltus" model (Mislevy & Wilson, 1995; Wilson, 1989; Wilson & Draney, 1995). In this presentation, we will introduce the Saltus model as an extension of the Rasch model, then briefly describe its application to the three data sets presented in the previous three papers. A discussion of future prospects for this approach concludes the presentation.

References

Mislevy, R. J., & Verhelst, N. (1990). Modeling item responses when different subjects employ different solution strategies. Psychometrika 55, 195-215.

Mislevy, R. J., & Wilson, M. (1995). Marginal maximum likelihood estimation for a psychometric model of discontinuous development. Psychometrika.

Wilson, M. (1989). Saltus: A psychometric model of discontinuity in cognitive development. Psychological Bulletin, 105, 276-289.

Wilson, M. & Draney, K. (1995, April). Partial credit in a developmental context: A mixture model approach. Paper presented at the annual meeting of the National Council for Measurement in Education, San Franscisco.