INTERNATIONAL RESEARCH GROUP MEASURING DEVELOPMENT

JPS 1994, CHICAGO
Symposium

Rasch Analysis Applied to Piagetian Theory:
Rethinking Statistical Methodology

Symposium Organizer & Chair: Trevor G. Bond, School of Education, James Cook University Australia

Invited Discussant: Ben Wright, MESA, University of Chicago

• Symposium Abstract
  
  
• Rasch analysis applied to Piagetian-type problems
  Gérald Noelting, Jean-Pierre Rousseau, Gino Coudé, Université Laval, Québec
  
  
• Quantitative assessment of the méthode clinique
  Trevor G. Bond & Erin Bunting, James Cook University
  
  
• Rasch analysis of a deductive reasoning task
  Ulrich Mueller, Kelly Reene, Willis F. Overton, Temple University

JPS 1995, BERKELEY
Symposium

Applying Rasch Analysis to Cognitive Developmental Data

Organizer and Chair, Trevor Bond, School of Education, James Cook University

• Symposium Abstract
  
  
• Rasch analysis applied to multiple-domain tasks
  Gérald Noelting, Jean-Pierre Rousseau, Gino Coudé, Université Laval, Québec
  
  
• Rasch-analysis of two recursive thinking tasks
  Ulrich Mueller, Mary Winn & Willis F. Overton, Temple University
  
  
• Detecting Cognitive Development with the Rasch Model
  Trevor G. Bond, James Cook University of North Queensland, Australia
  
  

JPS 1996, PHILADELPHIA
Symposium

Rasch Analysis in Context

Organizer: Theo Dawson, Graduate School of Education,U. C. Berkeley

Invited Discussant Trevor Bond, James Cook University, Australia

• Moral Reasoning and Evaluative Reasoning about the Good Life-A RaschAnalysis of Armon's 13-Year Lifespan Investigation
  Theo Dawson, University of California at Berkeley
  
  
• Rasch Scaling of a Set of Piagetian-Based Written Problems Representing Different Forms of Thought and Different Logical Operations
  William M. Gray, University of Toledo
  
  
• Children's Construction of the Operation of Addition
  Betsey A, Grobecker, Auburn University, Alabama
  
  
• The Analysis of Developmental Tasks Requiring the Reconciliation of Quantitative and Qualitative Approaches
  Gérald Noelting, Gino Coudé, & Jean-Pierre Rousseau, Université Laval, Québec

Poster presentations

• Quantitative Analysis of the Méthode Clinique II: The Child's Conception of Area
  Trevor G Bond & Kellie Parkinson, James Cook University, Australia
  
  
• An Examination of Intraindividual and Interindividual Developmental Trends in Moral and Evaluative Reasoning
  Theo Dawson, University of California at Berkeley
  
  
• A comparison of Rasch scaling, image analysis and multiple hierarchical analysis of a set of Piagetian based written problems representing different forms of thought and logical operations
  William M Gray, Christine Fox, University of Toledo; Trevor Bond, James Cook University & Richard J Hofmann, Miami University

Abstracts

Moral Reasoning and Evaluative Reasoning about the Good Life-A Rasch Analysis of Armon's 13-Year Lifespan Investigation

This paper presents a Rasch analysis of Armon's (1984a, 1984b, 1993, 1995a; Armon & Dawson, 1995) longitudinal/cross sectional study of moral reasoning and evaluative reasoning about the good. Armon interviewed a total of 42 individuals (23 female, 19 male) ranging in age from 5, at the first time of testing, to 86, at the final time of testing. Interviews took place at four year intervals-in 1977, 1981, 1985, and 1989.

Both Armon, who examined evaluative reasoning about the good, and Kohlberg, who
examined moral reasoning, describe structural developmental sequences that meet Piagetian criteria for structured wholeness, sequentiality, qualitative differences between stages, and hierarchical integration (Armon 1984, Colby & Kohlberg, 1987). Moreover, stages in Armon's and Kohlberg's sequences reflect similar organization, such that each stage in Armon's sequence meets structural criteria similar to those of the stage with the same numerical assignment in Kohlberg's sequence.

To assess moral reasoning, form A (three dilemmas: Heinz, Judge, Joe) of the Standard
Issue Scoring Manual (Colby & Kohlberg, 1987) was used with all participants except three who were familiar with it. For these participants, Forms B (three dilemmas: Dr. Jefferson, Judge, Judy) or C (three dilemmas: Korean, Valjean, Karl) were used. The same form was used with each participant at each time. Armon's (1984) Good Life interview was also administered at each test-time. The Good Life interview includes open-ended questions about the good life, good work, good friendship, and the good person. Interviews were recorded and transcribed for scoring. For each of the five domains studied, including the good life, good work, good friendship, the good person, and moral reasoning, participants were given general stage scores ranging from 1 to 5 in half-stage increments.

The results of the analysis support Armon's and Kohlberg's theoretical postulates. The
finding that persons think similarly across related domains, as demonstrated by good item fit and the tendency of levels of items with the same numerical value to cluster on the same range of the logit scale, supports the structured whole criterion. Moreover, the steady rise of mean estimates over the four test-times and the lack of statistically significant reversals supports claims for invariant sequence of development (Armon 1984, Colby & Kohlberg, 1987). Finally, it appears that stages in Armon's and Kohlberg's sequences reflect similar organization, in that each level of moral reasoning is found in the same logit range as each level of evaluative reasoning, a result that would not be expected if Armon's sequence failed to meet structural criteria similar to those of the stages in Kohlberg's sequence. The implications of these and other findings are elaborated.

Rasch Scaling of a Set of Piagetian-Based Written Problems Representing Different Forms of Thought
and Different Logical Operations

An underlying assumption within Piaget's approach to development is that there is a unidimensionality to the increasing differentiation and hierarchical organization of continual adaptations that are reflected in what Piaget labeled as different forms (i.e., stages) of thought. In essence, Piaget's focus and emphasis was on categorizing different forms of thought in terms of their complexity within the phenomenon of development. Obviously, the focus of those replicating and extending Piaget's work generally has been different than his focus on the development of forms of thought. Part of the problem has been the use of statistics that are not necessarily as appropriate as they could be, and should be, when evaluating developmental phenomena. Rasch analysis has provided an alternative means for evaluating Piaget's theory.

Subjects were 577 junior high school students who participated in an innovative science curriculum. On two successive days, subjects solved 24 written
problems (two forms of a written test of cognitive development with 12 problems on each form) that represented different concrete and formal operations, including the concrete operations of seriation, one-to-one multiplication of classes, one-to-many multiplication of classes, and addition of asymmetrical relations, as well as the formal operations of exclusion, systematic thinking, and proportional reasoning.

Rasch scaling of the levels within the problems clearly indicated that (a) within a logical operation, levels were scaled in their appropriate theoretical positions and (b) across logical operations, theoretically comparable levels were scaled in their appropriate theoretical positions. Scaling within logical operations and across logical operations provides strong psychometric support for Piaget's classification of forms and levels of thought along a single growth continuum.

Children's Construction of the Operation of Addition

Piaget argued that although young children actively experiment with number symbols and their respective properties, number is intelligible only when the synthesis of class (elements perceived as interchangeable units) and seriation (relative position in the order of correspondence) is achieved. This position was challenged by recent research that found very young children capable of transforming object sets in a nonverbal task situation. Research investigating the qualitative nature of the reasoning processes used in this "nonverbal" task could help to clarify these findings.

Six-to eight year old students who were judged to be of average ability by their teachers (n=42) participated in the study. Children's construction of operational structures was measured in a task assessing associativity of length (Piaget, 1987). This task involved setting up a series of string fences on a board, and asking participants to make several comparisons between them based on spatial position and relative distances. Six different problems were presented, and responses were scored according to operative level achieved and response correctness.

The nonverbal calculation task (Levine, Jordan, Huttenlocher, 1992) consisted of fourteen problems that required children to match the number disks placed under and on top of a mat by an investigator with an equal number of their own disks. In contrast to the task presented by Levine et al. (1992), the children were also asked to justify their choices. Fourteen number problems were presented (7 under/over 10) and scored in relation to five observed strategies as well as response correctness.

The same fourteen number problems used in the above tasks were placed on cards and held in front of the children until an answer was stated. Explanations for problem answers were required and any observed body behaviors noted. Responses were scored in relation to six observed strategies as well as response correctness.

Rasch analysis of the data revealed that for both the Piagetian tasks and the Levine tests, it was much easier for children to make correct judgments than it was for them to use operational justifications/strategies. The analysis revealed the sequential acquisition of abilities required by each of the tasks and, moreover, shows the developmental relationship between the complete solution of the Levine tests and the acquisition of concrete operational ability as elicited by the Piagetian tasks. Implications for testing, data analysis, and the growth of children's numerical ability are canvassed.

The Analysis of Developmental Tasks Requiring the Reconciliation of Quantitative and Qualitative Approaches

Discontinuity in cognitive development is now widely considered, although the content of successive structures remains obscure. This comes from the difficulty of finding criteria which are, at the same time, sufficiently general to cover a number of cognitive domains, and precise enough to characterize specific stages in the tasks themselves. Moreover, reconciliation of the structural analysis of contents and their statistical evidence is difficult to make. A method is necessary that allows the establishment of hierarchies, leading to a comparison of levels of difficulty within and between tasks. Moreover, this must show to what extent the hierarchy obtained meets, from its unidimensional character, the criteria of "structure d'ensemble." Rasch analysis represents a simple method which meets these goals.

We explore these theoretical issues in a developmental study bearing on different tasks, where common structural criteria have already been defined. Three instruments are submitted in individual questioning to 117 subjects between 10 and 22 years. Instruments are Mixing Juices (MJ), bearing on the development of rational number, made up of 14 items that range from intuitive to concrete to the post-formal stage; Coded Orthogonal Views (COV), bearing on the coding of polycubical objects, consisting of 12 items ranging from intuitive to post-formal; and Caskets Task text form (CKT), bearing on propositional reasoning, made up of 17 items ranging from intuitive to postformal. Items are scored pass or fail, and results of all three instruments are submitted together in a comprehensive Rasch analysis.

The Rasch procedure orders items according to logit measures. The resulting sequence is then analyzed qualitatively, giving five blocks of increasing difficulty, interpreted in the framework of Piagetian theory, with characteristic gaps between logit measures that correspond to a change in structure of the underlying concept. Investigation of fit shows that items of all three tests are on a same cognitive dimension, with good fit. At the individual level, subjects of a stage solve some items of the stage, but not of the following. At the statistical level, however, all items of a stage are grouped, whatever the tasks to which they belong. This is consistent with Piaget's concept of "structure d'ensemble." Rasch analysis shows the coherence of the stage concept, independent of the nature of the task.

An Examination of Intraindividual and Interindividual Developmental Trends in Moral and Evaluative Reasoning

Armon's 13-year lifespan study of the development of moral reasoning and evaluative
reasoning about the good is a rich source of data about the structural developmental process and its relationships with age, education, and gender. In this poster, regression-based growth modeling and Rasch analysis are combined to illuminate these relationships by tracing both intraindividual and interindividual patterns of growth. Results of the cross-sectional analysis show a moderate curvilinear relationship between age and development and a strong linear relationship between education and development, with small but statistically significant differences between men and women in adulthood and old age. The developmental analysis traces individual trajectories of development and compares these across age groups. Case studies of individual participants suggest that variation in interindividual developmental progress, which increases over the course of the lifespan, may be related to both dispositional and experiential factors.

Quantitative Analysis of the Méthode Clinique II: The Child's Conception of Area

This poster provides a detailed quantitaive analysis of the development of the concept of surface area of planee figures outlined by Piaget, Inhelder and Szeminska (1960) in The Child's Conception of Geometry. Forty -two primary school students were examined using the Piagetian technique, the
Méthode Clinique across four tasks: a conservation of area task involving transformation, the well-known meadows and farmhouse experiment as well as measurement of area by unit iteration and measurement of area by superposition tasks. The Partial Credit model of the Rasch family od latent trait models was used to analyse the subsequent data following the procedures adopted by Bond & Bunting (1995). For each of the tasks the developmental progression from early preoperational to late concrete operational ability is displayed. The consistencies across tasks of substage allocations are revealed and the nature and extent of horizontal décalage between the tasks is displayed in quantitative terms. The poster outlines the consistencies between the quantitative results of this investigation and predictions derived from the Piagetian oeuvre.

Nature of the Problem

Piaget's theory of intellectual development has serious implications in the field of education. Piaget's discoveries have had an impact upon curriculum, instruction and assessment in schooling, particularly for mathematics.The aim of this study is to determine what children understand about area as indicated by their interactions and comments using concrete materials in an interview situation and compare this to how they perform in written tasks. It is evident that the work of Piaget, Inhelder and Szeminska (1960) in The Child's Conception of Geometry, has influenced the developers of the mathematics curriculum in relation to understanding area concepts for primary students. With this knowledge, clinical interviews derived from Piagetian investigations were administered to determine the children's understanding of area. The aim was to discover what children's thoughts about area were and what misconceptions they might have.

Outline of the Analysis

This research was conducted at a small private school in northern
Australia. Forty-two students were randomly selected from alphabetical class lists from a total student population of 142 students. The distribution of the random sample of students included a fairly even spread of students from each grade. At the time of data collection the students ranged in age from 5 years; 3 months to 13 years; 1 month.

Tasks

The tasks used to investigate the students' understanding of area were the following, extracted from The Child's Conception of Geometry, Chapter 11:

Conservation of Area Tasks:

Task One - Transformation Tasks (consisting of 3 tasks)

Task Two - Meadows and Farmhouse Experiment

Task Three - Measurement by Unit Iteration

Task Four- Measurement by Superposition

(Piaget, Inhelder and Szeminska, 1960, pp. 261 -301)

A comparison of Rasch scaling, image analysis and multiple hierarchical analysis of a set of Piagetian based written problems representing different forms of thought and logical operations

Purpose

Conduct a preliminary analysis to determine which of three statistical techniques is best at representing the concrete operations and formal operations built into 24 written items. Harris' version of image factor analysis, the partial credit approach to Rasch analysis, and Hofmann's approach to multiple Guttman scaling were used.

Sample

577 seventh and eighth grade students in an experimental science education program completed both forms of Gray's How Is Your Logic?. The original sample was sorted in ascending order by age and assigned ID numbers. Students with odd numbered IDs were placed in the odd sample (N = 289) and students with even numbered IDs were placed in the even sample (N = 288). Table 1 describes the distribution of concrete and formal operations measured by the 24 items. Table 2 provides the demographic characteristics of both samples as well as Kaiser-Meyer-Olkin measures of sample adequacy.

Harris Image Analysis

Within each sample, Cattell's scree test, eigenvalue > 1.00 rule, Hofmann's average complexity of various solutions, and the interpretability of factor solutions each suggested different numbers of factors to extract. Across samples, each criterion was consistent in suggesting the same number of factors to extract (e.g., the eigenvalue >1.00 rule suggested that six factors should be extracted in each sample). Factors representing Proportional Reasoning and Addition of Asymmetrical Relations were the most stable across samples and across solutions which varied in number of factors extracted. Factors representing Make Correct Inclusion and Deny Incorrect Inclusions were next most stable. Factors representing Combinatorial Thought and Multiplication of Classes were the least stable.

Partial Credit Rasch Analysis

Person separation was above the necessary 2.0 for each analysis (2.42 for the even and 2.06 for the odd). Item separation was excellent for both analyses (10.53 for even & 9.22 for odd). Table 3 and Figure 1 provide the results for the odd sample and Table 4 and Figure 2 provide the results for the even sample. Figure 3 provides a joint plot of the measures from both samples. With the exception of three items, the interpretation of the underlying variable was the same for both analyses. In the analysis of the odd set, three concrete items (A7, B5, & A4) were misplaced into the more difficult region of the continuum (dominated mostly by items measuring formal thought). This is illustrated in both the person-variable maps for each analysis and the plot of the separate item calibrations. This misplacement of the three items is most likely due to some aberrant responses to the questions or improper judging of the responses, as indicated by the number of person misfits for each analysis (34 person misfits in the even set and 28 person misfits in the odd set). Elimination of these responses should result in a more consistent variable across the two halves of the data set.

Multiple Guttman Scaling

The final product of an exploratory multiple Guttman scale analysis is the scale dendogram. The scale dendogram is a graphic representation of all of the scales and the items defining them. The major intention of the scale dendogram is to help conceptualize visually the scale structure of the data analyzed. The dendogram for the odd sample is presented in Figure 4 and the even sample is presented in Figure 5. When comparing the dendograms it is apparent that there are key items linked to each other that form a common basis for a number of scales, several of which are common to both the odd and even samples. There are no particularly large scales that are common to both samples. The largest common scale is defined by items (A4, B5, B1, A5). These four items are in part common to seven of the eight scales defined by the odd sample and common to eight of the nine scales defined by the even sample. Both data samples defined the same non-scalar items. Three of the doublets are common to both samples, and four items define a scale that is common to fifteen of the seventeen scales defined by the two samples (A4, B5, B1, A5). From a hierarchical perspective, it appears that How Is Your Logic? has too many similar items and not enough heterogeneity of item difficulty.

Applying Rasch Analysis to Cognitive Developmental Data:
Abstract Summary

Rasch analysis applied to multiple-domain tasks

Rasch-analysis of two recursive thinking tasks

The present study examined the dimensionality of two measures assessing different levels of recursive thinking. The first measure was a task designed by Miller (Miller, Kessel & Flavell, 1970). This task consists of 18 items which make up four different levels of recursive thought, with third order thought (thinking about thinking about thinking) being the highest level. The second measure was based on subjects' responses to a short video tape in which two individuals discussed experiences of self reflection. Subjects were asked to identify with one of the individuals and to justify their selection. Three items corresponding to different levels of self reflection were constructed on the basis of the subjects' answers and justifications. Subjects were 22 5th-, 21 8th-, and 30 11th-graders. Rasch analysis of Miller's recursive thinking task showed that 17 out of the 18 items fitted the Rasch model. Consistent with Miller's (Miller, Kessel & Flavell, 1970) study, four levels of recursive thought were found. Some items, however, were not distributed as expected. Rasch analysis also proved useful in helping to decide how to score ambiguous responses. Finally, Rasch analysis of the joint fit of the two measures of recursive thinking showed that the 3 items of the second measure did not fit the Rasch model.

Detecting Cognitive Development with the Rasch Model

It is unfortunate that most of what passes for developmental research is restricted to cross-sectional design and it is difficult to infer what it is that develops. To elucidate this problem with regard to Piagetian cognitive development two samples of secondary school students were assessed on a validated pencil and paper measure of formal operational thinking. This assessment was repeated after one year with the first sample and after two years with the second sample. Traditional statistical indices show the robust reliability of the test and affirm that statistically significant development had taken place over the intervening periods. More interesting however is the light that Rasch analysis shines on these data. It identifies the subjects for whom growth did or did not take place as well as providing corroborating evidence for the account of formal operational thought provided by Inhelder and Piaget (1955/58).

Rasch Analysis Applied to Piagetian Theory:
Rethinking Statistical Methodology
Symposium ABSTRACT

Rasch analysis applied to Piagetian-type problems

Rasch analysis is applied to Piagetian-type problems in two different situations. A first experiment involves a comparison between clinical and group interrogation. A logical task is devised from a mathematical problem submitted by Smullyan (1978). The subject must deduce, from inscriptions on various caskets and accompanying truth-values, where a portrait is hidden. A graphic form of the task is constructed for younger subjects.

The same graphic test, made up of 18 items, is submitted to two groups, one of 35 subjects between 4 and 9 years questioned clinically, the other 64 subjects in grades 1 to 3. Rasch analysis is applied to both series of results. The same order is obtained for item difficulty in both procedures. But while clinical interrogation distributes subjects and items evenly along the scale, group interrogation creates a large gap between level 1 items (preoperational) and level 2 items (concrete operational). Twenty-two subjects were situated within this gap, at a level where no items exist. Clinical interrogation leads to a better fit between ability and item. Group questioning at young ages does not lead to performance in accord with ability. This leads to trying to interpret the meaning of such gaps in a Rasch order of item difficulty. Do they correspond to discontinuity in the underlying ability?

A second experiment involves the comparison of two Piagetian-type tasks submitted in group form to 120 subjects from grades 4, 6, 8, 11 and University. The first task involves proportional reasoning, the second geometrical projection. Group procedures are elaborated for both. Rasch analysis applied separately to results of each task yield an order of items consistent with levels anticipated. When results are grouped in a single analysis, items of corresponding levels of each task cluster together, with gaps between levels. This strongly suggests the existence of structures d'ensemble common to two domains. Four clusters are obtained corresponding to preoperations, concrete, transitional and formal operations. However, Rasch does not separate these items of different type except as concerns their level of difficulty. Would a supplementary kind of analysis be necessary to differentiate between levels? Or is qualitative analysis necessary to distinguish between items of different type?

Quantitative assessment of the méthode clinique

Central to this investigation has been the collection of 58 méthode clinique protocols recording the attempts of a sample of Australian adolescents to solve the pendulum problem from Inhelder & Piaget's, The Growth of Logical Thinking. The performances were given substage allocations (IIA, IIB, IIIA, IIIB) according to the principles of qualitative and logico-mathematical analysis delineated by Piaget in Chapter 4 of that text. Subsequently, a highly detailed description of the abilities delineated in that chapter allowed the development of a set of 34 performance criteria which were used to quantitatively score the protocols.

The results of rating scale analysis, a variation of Rasch analysis, provided a fine grained description of these abilities not previously countenanced under qualitative or quantiative analytical techniques. The results provide substantiation of key constructs from Piagetian theory but raise the question of how Rasch principles of test item banking may be applied to these and other Australian data to develop a comprehensive description of formal operational thinking performance for large groups across many such tasks.

Rasch-Analysis of a deductive reasoning task

The purpose of this study was to examine the relationship between self-concept development and the development of cognitive functioning. As an indicator of level of cognitive functioning Overton's version of Wason's selection task was used. This measure focusses on the deductive reasoning competence. Previous research it has been shown that Overton's version of the selection task captures the development of reasoning competence in childhood and adolescence. In the selection task, a conditional rule ("if p then q"), together with four cards ("p", "-p", "q", "-q") is presented. The subject's task is to decide which card(s) must be turned over in order to assess whether the rule is being violated. Eleven selection problems with familiar content (10 test and one warm-up problem with feedback) were used.

Subjects were 258 7th- (90), 8th- (106) 9th-grader (62). Range of solution rates for the 10 problems were from .39 to .75. The correlation matrix (Phi-correlations) of the items were factor analyzed (PCA); two factors emerged (Eigenvalue criterion). All items loaded highly on the first factor, three items were loaded highly on the second factor (negative correlation). A subsequent Rasch analysis demonstrated that the construct measured by the items was unidimensional. All 10 items fitted the Rasch model, implying that all test items have equal discriminating power and measure the same underlying ability. The Rasch-analysis clarified the fact that the three items which loaded on the second factor were the most difficult items.

Rasch analysis indicates that the items can be grouped into three clusters with different difficulty. This leads to the speculation that different levels of deductive reasoning (concrete operational, transitional, formal operational) are related to differential problem solving. These results raise the question as to what extent and in what way could Rasch-analysis facilitate the further discrimination between Piagetian levels of reasoning.