Human Observation for Developing Sensitive Learning Environments
Human behavior in interactive learning environments can manifest itself in a
number of ways. Even though a human being watching a student can often quickly and easily identify characteristic patterns of behavior such as gaming the system, it can be difficult to encode that knowledge into a rational, knowledge-engineering model.
In my research, I attempt to encode human beings' ability to distinguish behaviors into interactive learning environments, using a combination of human observational data and data mining and machine learning.
I have explored three observational techniques.
The first, qualitative field observations, consists of simply watching students in real settings of use (for example, classrooms) as they use the educational software, and noting down every behavior the students engage in. This enables the researcher to know what types of behavior occur, but not their prevalence or impact on learning. It is important to conduct these observations in real settings of use, to determine what behaviors really occur (for instance, students are more likely to talk off-task in the field than in a laboratory setting).
The second, quantitative field observations, consists of systematically observing students in real settings of use, with a pre-determined coding scheme (usually developed through qualitative observations). Observations are conducted in a very systematic fashion, with each observation a pre-selected duration, observations taken via peripheral vision, and only one student observed at a time. When combined with pre-test and post-test data, quantitative observations can determine different behaviors' prevalence and relationship to learning gains. However, quantitative field observations are very time-consuming to conduct, and can not be conducted retrospectively.
The third, text action description observations, consist of pretty-printing segments of student action within an interactive learning environment and making judgments about the student's behavior during those segments. Text action descriptions have been shown to have lower inter-rater reliability than live quantitative field observations, but to correlate equally well to gold-standard measures of student behavior as live observations do, for some categories of behavior. Text action description observations are very quick to conduct, and can be conducted retrospectively on data that is years old. However, they are only as reliable as the logs they are drawn from, and cannot capture any behavior which has an offline component (such as talking off-task).
Ido Roll has compared machine learning models of student behavior developed using human observations to models developed using knowledge-engineering. He found that the models developed using human observations were more precise, but less easy to generalize; he also found that each model captured important aspects of student behavior that the other model did not capture.
Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students
"Game The System". Proceedings of ACM CHI 2004: Computer-Human
Interaction, 383-390. [pdf]
Roll, R., Baker, R., Aleven, V., McLaren, B., Koedinger, K. (2005) Modeling Students' Metacognitive Errors in Two Intelligent Tutoring Systems. Proceedings of User Modeling 2005, 367-376.
Baker, R.S.J.d., Corbett, A.T., Wagner, A.Z. (2006) Human Classification of Low-Fidelity Replays of Student Actions. Proceedings of the Educational Data Mining Workshop at the 8th International Conference on Intelligent Tutoring Systems, 29-36. [pdf]
Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sevilla, L.R.S., Sugay, J.O., Tep, S., Viehland, N.J.B. (2007) Affect and Usage Choices in Simulation Problem Solving Environments. Proceedings of Artificial Intelligence in Education 2007, 145-152. [pdf]
Collaborators and co-authors