Competence Assessment in Education - Research, Models and Instruments

Contents

Chapter 1: Competence Assessment in Education: An Introduction

1.1 The German DFG-Priority Program “Competence Models for Assessing Individual Learning Outcomes and Evaluating Educational Processes”

1.2 Research Areas of the DFG-Priority Program

1.2.1 Cognitive Modeling and Assessment of Competencies

1.2.2 Innovations in Psychometric Models and Computer-­Based Assessment

1.2.3 Reception and Usage of Assessment Results

1.3 Conclusion

References

Part I: Modeling and Assessing Student Competencies

Chapter 2: Science-P I: Modeling Conceptual Understanding in Primary School

2.1 The Assessment of Science Competency in Primary School

2.2 Modeling Conceptual Understanding in Primary School Science

2.2.1 Model Specification and Item Construction

2.2.2 Conceptual Understanding: Dimensions and Levels

2.2.3 Validity

2.3 The Development of Conceptual Understanding in Primary School Science

2.4 Conceptual Understanding and Scientific Reasoning

References

Chapter 3: Science-P II: Modeling Scientific Reasoning in Primary School

3.1 Science-P

3.2 Development of Our Inventory

3.3 Convergent Validity of Paper-and-Pencil Inventory and Interviews

3.3.1 Method

3.3.1.1 Participants

3.3.1.2 Material

3.3.1.3 Procedure

3.3.2 Results

3.3.2.1 Pre-analyses

3.3.2.2 Convergent Validity

3.3.3 Discussion

3.4 Scientific Reasoning: Development from Grades 2 to 4

3.5 Competence-Structure Model of Scientific Reasoning: Hierarchical Levels of Competence

3.6 Outlook

References

Chapter 4: The Heidelberg Inventory of Geographic System Competency Model

4.1 The Role of Geographic System Competency in Geography Education

4.2 Study Overview

4.3 CogLabs

4.3.1 Description of the Measurement Instruments

4.3.2 Selected Results

4.4 First Quantitative Study (Q1)

4.4.1 Description of the Measurement Instruments

4.4.2 Dimensions of the Competency Model

4.4.3 Levels of the Competency Model

4.5 Second Quantitative Study (Q2)

4.5.1 Description of the Measurement Instruments

4.5.2 Dimensions of the Competency Model

4.5.3 Levels of the Competency Model

4.6 Discussion

4.6.1 Dimensions of GSC

4.6.2 GSC Levels

4.7 Conclusions

References

Chapter 5: An Extended Model of Literary Literacy

5.1 The Comprehension of Literary and Expository Texts

5.2 Current Research on Literary Literacy and Further Directions

5.2.1 The Internal Structure of Literary Literacy

5.2.2 The Need for an Extended Model of Literary Literacy

5.3 Research Objectives

5.4 Method

5.4.1 Sample

5.4.2 Measures

5.4.2.1 Semantic Literary Literacy

5.4.2.2 Idiolectal Literary Literacy

5.4.2.3 The Ability to Recognize Foregrounded Passages (Foregrounding)

5.4.2.4 Specific Literary Knowledge

5.4.2.5 The Ability to Recognize Emotions Intended by a Literary Text

5.4.3 Statistical Analyses

5.5 Results

5.6 Discussion

5.6.1 The Structure of Literary Literacy

5.6.2 Some Considerations on Teaching Literary Literacy

5.6.3 Limitations of the Study

References

Chapter 6: Self-Regulated Learning with Expository Texts as a Competence: Competence Structure and Competence Training

6.1 Theoretical Background

6.1.1 Integrated Model of Self-Regulated Learning

6.2 Research Questions and Hypotheses

6.3 Studies 1a and 1b: A “Component Perspective” on the Structure of Self-Regulated Learning Competence

6.4 Study 2: A “Process Perspective” on the Structure of Self-Regulated Learning Competence

6.5 Discussion

References

Part II: Modeling and Assessing Teacher Competencies

Chapter 7: Investigating Pre-service Teachers’ Professional Vision Within University-Based Teacher Education

7.1 Introduction

7.2 Modeling the Structure of Professional Vision

7.2.1 Noticing: Selective Attention to Important Classroom Events

7.2.2 Reasoning: Interpretation of Important Classroom Events

7.3 Testing the Structure of Professional Vision

7.3.1 The Assessment Tool Observer

7.3.2 Interrelation Between the Three Reasoning Dimensions

7.4 Investigating Changes in Professional Vision Within University-Based Teacher Education

7.4.1 The Role of Formal and Informal OTL

7.4.2 The Design of Formal and Informal OTL

7.5 Building the Bridge from Professional Vision to Teaching Action

7.5.1 M-Teach Events as Assessment of Teaching Action

7.5.2 Pre-service Teachers’ Teaching Skills in M-Teach Events

7.6 Conclusion and Outlook

References

Chapter 8: Teacher Knowledge Experiment: Conditions of the Development of Pedagogical Content Knowledge

8.1 Introduction

8.1.1 The Construct of Pedagogical Content Knowledge

8.1.2 Conditions for the Development of Pedagogical Content Knowledge: The Role of Prior Content Knowledge and Pedagogical Knowledge

8.1.3 The Present Study

8.2 Methods

8.2.1 Participants

8.2.2 Treatments

8.2.3 Measures

8.2.4 Baseline Equivalence and Treatment Implementation Checks

8.3 Results

8.3.1 Measurement of Pre-service Teachers’ Knowledge

8.3.2 Testing the Assumptions on PCK Development

8.4 Discussion

References

Chapter 9: Teachers’ School Tracking Decisions

9.1 Introduction

9.2 Dual Process Models of Decision Making

9.3 The Adaptive Diagnostic Competency Model (ADCM)

9.4 Testing the ADCM

9.4.1 Student Case Vignettes (Pre-studies 1 and 2)

9.4.2 The ADCM: Case Consistency and Expertise (Study 1)

9.4.3 The ADCM: Case Consistency and Accountability (Study 2)

9.5 Training Study

9.6 Discussion

References

Chapter 10: Modeling, Measuring, and Training Teachers’ Counseling and Diagnostic Competencies

10.1 Introduction

10.2 Project Goals

10.3 Modeling Teachers’ Counseling and Diagnostic Competencies

10.3.1 Theoretical Background

10.3.2 Method

10.3.3 Results

10.3.4 Relationship Between Teachers’ Counseling and Diagnostic Competence

10.4 Measuring Teachers’ Counseling and Diagnostic Competence

10.4.1 Scenario Tests

10.4.2 Situational Judgment Test

10.4.3 Knowledge Tests and Self-Assessment Questionnaires

10.5 Training Teachers’ Counseling and Diagnostic Competence

10.5.1 Training Program in Diagnostic Competence for In-Service Teachers

10.5.2 Training Programs in Counseling and Diagnostic Competence for Prospective Teachers

10.6 Conclusions and Outlook

References

Chapter 11: Development and Evaluation of a Competence Model for Teaching Integrative Processing of Texts and Pictures (BiTe)

11.1 The “BiTe-Project”

11.2 Theoretical Background

11.2.1 Challenges of Picture-Text-Integration (PTI)

11.2.2 Teachers’ Competencies for Teaching the Integrative Processing of Pictures and Texts

11.2.3 Quality of Instruction

11.3 Research Questions and Hypotheses

11.4 Methods

11.4.1 Sample and Study Design

11.4.2 Measures

11.5 Results

11.5.1 Research Question (1): Knowledge About PTI

11.5.2 Research Question (2): Teachers’ Attitudes, Motivation, and Self-Related Cognitions Towards PTI and Diagnostics in PTI

11.5.3 Research Question (3a): Teachers’ Accuracy of Judgment

11.5.4 Research Question (3b): Teachers’ Accuracy of Judgment, Knowledge, and Duration of Contact

11.5.5 Research Question (4a): Relations Between Teachers’ Competencies and Instruction

11.5.6 Research Question (4b): Relations Between Teachers’ Competencies and Students’ Competence and Engagement

11.6 Discussion

References

Part III: Modeling and Assessing Vocational Competencies and Adult Learning

Chapter 12: Multidimensional Competency Assessments and Structures in VET

12.1 Introduction

12.2 Theoretical Issues and Operationalization

12.3 Research Questions

12.4 Research Design

12.5 Hypotheses and Results

12.5.1 Research Question 1: Competency Structures Within the Construct of Not Directly Action-Centered Occupation-Specific Knowledge: Dimensionality and Its Development

12.5.2 Research Question 1: Competency Structures Within the Construct of Action-Centered Occupation-­Specific Knowledge: Dimensionality and Its Development

12.5.3 Preliminary Analysis for Research Question 2: Construct of Occupation-Specific Problem Solving: Validity and Reliability

12.5.4 Research Question 2: Competency Structures Between Different Constructs

12.6 Summary and Final Discussion

References

Chapter 13: Professional Competencies of Building Trade Apprentices After Their First Year of Training

13.1 Introduction

13.2 Aims and Objectives

13.3 Vocational Training in the Building Trades

13.4 Professional Competence of Building Trade Apprentices

13.5 Research Design and Data Collection

13.6 Results

13.6.1 Sociodemographic Factors, Cognitive Abilities and Apprentices’ Performance at the Beginning of the First Training Year

13.6.2 Professional Competence at the End of the First Training Year

13.7 Additional Findings and Prospects

References

Chapter 14: Assessing Tomorrow’s Potential: A Competence Measuring Approach in Vocational Education and Training

14.1 Background

14.1.1 Prospects and Demand for Adequate Competence Assessments in Vocational Education

14.1.2 The Original Conceptualization of Final Examinations in the Area of Business and Commerce

14.1.3 Assessment Model for Commercial Vocations

14.2 Method

14.2.1 Sample

14.2.2 Examination of Validity

14.2.3 Examination of Reliability

14.3 Results

14.3.1 Results for the Test’s Validity

14.3.2 Results for the Test’s Reliability

14.4 Discussion

14.5 Conclusions

Appendix

Ceraforma Keramik AG

Business Process 1

References

Part IV: Competency Development: Modeling of Change and Training of Competencies

Chapter 15: The Development of Students’ Physics Competence in Middle School

15.1 Introduction

15.2 Theoretical Background

15.2.1 Students’ Understanding of Energy

15.2.2 Students’ Learning About Energy

15.3 Research Questions

15.4 Project Design

15.5 Phase 1: The Cross-Sectional Study

15.5.1 Method

15.5.2 Results

15.6 Phase 2: Longitudinal Study

15.6.1 Method

15.6.2 Results

15.7 Summary and Outlook

References

Chapter 16: Modeling and Fostering Decision-Making Competencies Regarding Challenging Issues of Sustainable Development

16.1 Introduction

16.2 A Competence Model for Decision Making with Respect to Sustainable Development

16.3 Measurement Instruments and Competence Modeling

16.3.1 “Developing Solutions”: Development of the Measurement Instrument

16.3.1.1 Sample

16.3.1.2 Measures: Tasks and Items

16.3.1.3 Instrument Functioning

16.3.2 Modeling of “Developing Solutions”

16.3.3 Discussion

16.4 Experimental Validation: A Comparison of Socioscientific Decision Making with Analytical Problem Solving

16.4.1 Objectives and Research Design

16.4.2 Methods

16.4.2.1 Participating Students and Teachers

16.4.2.2 Trainings and Learning Material

16.4.2.3 Measures

16.4.3 Results of the Pilot Study

16.4.4 Discussion

16.5 Conclusions and Outlook

References

Chapter 17: Metacognitive Knowledge in Secondary School Students: Assessment, Structure, and Developmental Change

17.1 Theoretical Background

17.1.1 Methodological Issues Regarding the Assessment of Declarative Metacognitive Knowledge

17.1.2 Design of the EWIKO Study

17.1.3 Test Instruments

17.2 Overview of Major Results

17.2.1 Development of Metacognitive Knowledge: Sources of Interindividual Differences

17.2.2 Domain-Specificity—A Transitional Period of Metacognitive Development?

17.2.3 Interrelations Between Metacognitive Knowledge and Achievement

17.3 Discussion

References

Chapter 18: Development of Dynamic Usage of Strategies for Integrating Text and Picture Information in Secondary Schools

18.1 Texts Combined with Instructional Pictures

18.2 Theoretical Background

18.2.1 Taxonomies of Text-Picture-Integration

18.2.2 Strategies for Integrative Processing of Text and Pictures

18.3 Research Questions and Hypotheses

18.4 Method

18.5 Results

18.5.1 Reading and Observation Times

18.5.2 Transitions Between Texts, Pictures, and Items

18.6 Discussion

References

Chapter 19: Training in Components of Problem-Solving Competence: An Experimental Study of Aspects of the Cognitive Potential Exploitation Hypothesis

19.1 Introduction

19.2 Theoretical Framework

19.2.1 Problem Solving in PISA 2003: The Cognitive Potential Exploitation Hypothesis

19.2.2 Components of Problem-Solving Competence

19.3 Research Questions

19.4 Study I

19.4.1 Methods

19.4.2 Results

19.4.3 Discussion

19.5 Study II

19.5.1 Methods

19.5.1.1 Data Analysis

19.5.2 Results

19.5.2.1 Planning

19.5.2.2 Problem Solving

19.5.2.3 Mathematics

19.5.3 Discussion

19.6 General Discussion

19.6.1 Limitations and Future Research

References

Chapter 20: An Intensive Longitudinal Study of the Development of Student Achievement over Two Years (LUISE)

20.1 Introduction

20.2 Student Achievement in Languages

20.3 Student Achievement in Mathematics

20.4 Changes in Student Achievement in Mathematics and Native Language

20.5 The Present Study

20.5.1 Method

20.5.1.1 Procedure and Participants

20.5.1.2 Measures

20.5.1.2.1 German Achievement Tests

20.5.1.2.2 Mathematics Achievement Tests

20.5.2 Results

20.5.2.1 German Achievement

20.5.2.2 Mathematics Achievement

20.6 Summary and Discussion

References

Part V: Innovations in Psychometric Models and Computer-Based Assessment

Chapter 21: Multidimensional Structures of Competencies: Focusing on Text Comprehension in English as a Foreign Language

21.1 Introduction

21.2 Test Development

21.2.1 Item Characteristics

21.2.2 Item Development

21.2.3 Validation of Item Characteristics

21.3 Test and Item Analysis

21.3.1 Sample and Data Collection

21.3.2 Unidimensional Test and Item Analysis

21.3.3 Item Difficulties Across Domains

21.3.4 Local Dependencies

21.3.5 Multidimensional Analysis

21.4 Discussion

21.4.1 Research Perspectives

21.4.2 Implications for Educational Contexts

References

Chapter 22: Multidimensional Adaptive Measurement of Competencies

22.1 Problem

22.2 Consideration of Multiple Constraints in MAT

22.2.1 Multidimensional Maximum Priority Index

22.2.2 Research Objective

22.2.3 Method

22.2.4 Procedure

22.2.5 Results

22.2.6 Discussion

22.3 Using Prior Information for Item Selection and Ability Estimation

22.3.1 Research Questions

22.3.2 Method

22.3.3 Results

22.3.4 Discussion

22.4 The Multidimensional Adaptive Testing Environment (MATE)

22.4.1 Computerization of Items

22.4.2 Assignment of Item Parameters

22.4.3 Configuration of Tests and Test Batteries

22.4.4 Pre-operational Simulation Studies

22.4.5 Graphical User Interface, System Requirements, Availability and Manual

22.5 Empirical Application

22.6 Conclusion

References

Chapter 23: Development, Validation, and Application of a Competence Model for Mathematical Problem Solving by Using and Translating Representations of Functions

23.1 Introduction

23.2 Construction of a Theoretical Framework Model

23.3 Development and Empirical Validation of Psychometric Models

23.3.1 Basic Model: Representations and Translations Between and Within Situational, Numerical, and Graphical Representation

23.3.2 Extension: Inclusion of Algebraic Representation/Cognitive Action

23.3.3 Extension: Hierarchical Models

23.4 Development and Evaluation of a Computerized Adaptive Testing Procedure

23.4.1 Aims for the Development of the Adaptive Test

23.4.2 Item Pool and Calibration

23.4.3 Pre-operational Simulation Study

23.5 Discussion

References

Chapter 24: Relating Product Data to Process Data from Computer-Based Competency Assessment

24.1 Introduction

24.2 Study 1: The Effect of Time on Task Success Differs Across Persons and Tasks

24.2.1 Research Goal and Hypotheses

24.2.2 Methods

24.2.3 Results

24.2.4 Discussion

24.3 Study 2: Benefits for Task Completion from the Automatization of Subtasks

24.3.1 Research Goal and Hypotheses

24.3.2 Methods

24.3.3 Results

24.3.4 Discussion

24.4 Study 3: Number of Interactions: More Is Not Always Better

24.4.1 Research Goal and Hypotheses

24.4.2 Methods

24.4.3 Results

24.4.4 Discussion

24.5 Study 4: Problem Solver Types: Different Ways to Success in Information Problems

24.5.1 Methods

24.5.2 Results

24.5.3 Discussion

24.6 How to Handle Unstructured Process Data?: The Log File Data Extraction Tool

24.7 Conclusions and Final Remarks

References

Chapter 25: Dynamic Problem Solving: Multiple-Item Testing Based on Minimally Complex Systems

25.1 Introduction

25.2 Modeling of Problem Solving Competencies

25.3 Development of Computer-Based Assessment Instruments

25.4 Empirical Tests of the Newly Developed Instruments

25.5 Educational Application: PISA 2012

25.5.1 Two Additional Issues: Optimization and Causal Diagrams

25.6 Future Developments

References

Part VI: Feedback From Competency Assessment: Concepts, Conditions and Consequences

Chapter 26: Formative Assessment in Mathematics Instruction: Theoretical Considerations and Empirical Results of the Co2CA Project

26.1 Formative Assessment: A Promising Approach to Improving Teaching and Learning?

26.1.1 Formative Assessment: State of the Art

26.1.2 Components of Formative Assessment

26.1.3 How Formative Assessment Affects Learning

26.2 The Four Studies of the Co2CA Project

26.2.1 Survey Study

26.2.2 Experimental Study

26.2.3 Intervention Study

26.2.3.1 Aims and Research Questions of the Intervention Study

26.2.3.2 Design of the Intervention Study

26.2.3.3 Selected Results of the Intervention Study

26.2.4 Transfer Study

26.2.4.1 Aims and Research Questions of the Transfer Study

26.2.4.2 Design of the Transfer Study

26.2.4.3 First Results of the Transfer Study

26.3 Summary

References

Chapter 27: Arguing Validity in Educational Assessment

27.1 Introduction

27.2 The Validity Concept

27.2.1 The Place of Standard Setting in Educational Assessment

27.3 The Argument Approach to Evaluating Validity

27.3.1 A Structured Validity Argument

27.3.1.1 Warrant of Well-Defined Content Standards

27.3.1.2 Warrant of Test Alignment to the Content Domain

27.3.1.3 Warrant of Well-Defined Performance Level Descriptors

27.3.1.4 Warrant of Reliable Test-Score Measurement

27.3.1.5 Warrant of Defensible Cut-Score Placements

27.4 Discussion

References

Chapter 28: Evaluating Prerequisites for the Development of a Dynamic Test of Reading Competence: Feedback Effects on Reading Comprehension in Children

28.1 Introduction

28.2 The Idea of Dynamic Assessments

28.3 Dynamic Assessments of Reading Competence: Existing Approaches and Challenges

28.4 Experiments on the Effectiveness of Feedback on  Reading Comprehension in a Train-Within-Test Setting

28.4.1 Experiment 1

28.4.2 Experiment 2

28.4.3 Experiment 3

28.5 Effects of Feedback on Reading Comprehension Within a Computer-Delivered Test: Lessons Learned

28.6 Prospects of Dynamic Tests of Reading Competence

References