Baker-Rodrigo Observation Method Protocol (BROMP)

BROMP is a protocol for Quantitative Field Observations (QFOs) of student affect and behavior. It has also been used to code teacher behaviors. BROMP is currently in version 1.0.

First developed by Ryan Baker and Ma. Mercedes T. Rodrigo, BROMP is a holistic coding procedure that has been used in thousands of hours of field observations of students from kindergarten to undergraduate populations. It has been used for several purposes, including to study the engagement of students participating in a range of classroom activities (both activities involving technology and more traditional classroom activities) and to obtain data for use in developing automated models of student engagement with Educational Data Mining (EDM).

A range of coding schemes have been used with the BROMP protocol. Students are observed individually, in a pre-determined sequence in order to avoid bias towards more interesting classroom activities. Observers have up to 20 seconds to determine which behavior and which affective state a student is exhibiting, but they record only the first of each.

Additional information about BROMP is available through our training manual:

Ocumpaugh, J., Baker, R.S.J.d., Rodrigo, M.M.T. (2012) Baker-Rodrigo Observation Method Protocol (BROMP) 1.0. Training Manual version 1.0. Technical Report. New York, NY: EdLab. Manila, Philippines: Ateneo Laboratory for the Learning Sciences. [pdf]

Using the BROMP coding scheme is facilitated by the HART Android app, available free of charge for non-commerical or academic use. HART can be obtained from this link or as part of the GIFT distribution from the U.S. Army Research Laboratory [link].

BROMP Certification

To date, 60 people have obtained official certification as BROMP-certified coders. The training to become a BROMP-certified coder takes approximately one day on-site, and BROMP-certified coders can train and certify other individuals. Achieving BROMP certification involves obtaining inter-rater agreement of over 0.6 (Cohen's Kappa) in field settings with a previously certified coder. For more information on arranging BROMP certification, please contact Ryan Baker.

Publications

BROMP has been used in many peer-reviewed publications. Some examples are given below.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students "Game The System". Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390. [pdf]

Baker, R.S., Corbett, A.T., Koedinger, K.R. (2004) Detecting Student Misuse of Intelligent Tutoring Systems. Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 531-540. [pdf]

Baker, R.S., Corbett, A., Koedinger, K., Roll, I. (2005) Detecting When Students Game The System, Across Tutor Subjects and Classroom Cohorts. Proceedings of User Modeling 2005, 220-224. [pdf]

Baker, R.S., Roll, I., Corbett, A.T., Koedinger, K.R. (2005) Do Performance Goals Lead Students to Game the System? Proceedings of the International Conference on Artificial Intelligence and Education (AIED2005), 57-64. [Finalist for Best Paper Award] [pdf]

Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Evenson, S.E., Roll, I., Wagner, A.Z., Naim, M., Raspat, J., Baker, D.J., Beck, J. (2006) Adapting to When Students Game an Intelligent Tutoring System. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 392-401. [Won Best Paper Award] [pdf]

Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Roll, I. (2006) Generalizing Detection of Gaming the System Across a Tutoring Curriculum. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 402-411. [Finalist for Best Paper Award] [pdf]

Baker, R.S.J.d. (2007) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068. [Honorable Mention for Best Paper Award] [pdf]

Baker, R.S.J.d., Rodrigo, M.M.T., Xolocotzin, U.E. (2007) The Dynamics of Affective Transitions in Simulation Problem-Solving Environments. Proceedings of the Second International Conference on Affective Computing and Intelligent Interaction . [pdf]

Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sevilla, L.R.S., Sugay, J.O., Tep, S., Viehland, N.J.B. (2007) Affect and Usage Choices in Simulation Problem Solving Environments. Proceedings of Artificial Intelligence in Education 2007, 145-152. [pdf]

Baker, R.S.J.d., Corbett, A.T., Roll, I., Koedinger, K.R. (2008) Developing a Generalizable Detector of When Students Game the System. User Modeling and User-Adapted Interaction, 18, 3, 287-314. [official pdf] [pdf]

Baker, R., Walonoski, J., Heffernan, N., Roll, I., Corbett, A., Koedinger, K. (2008) Why Students Engage in "Gaming the System" Behavior in Interactive Learning Environments. Journal of Interactive Learning Research, 19 (2), 185-224. [pdf]

Rodrigo, M.M.T., Anglo, E.A., Sugay, J.O., Baker, R.S.J.d. (2008) Use of Unsupervised Clustering to Characterize Learner Behaviors and Affective States while Using an Intelligent Tutoring System. Proceedings of International Conference on Computers in Education, 49-56.[pdf]

Rodrigo, M.M.T., Baker, R.S.J.d., d'Mello, S., Gonzalez, M.C.T., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sugay, J.O., Tep, S., Viehland, N.J.B. (2008) Comparing Learners' Affect While Using an Intelligent Tutoring Systems and a Simulation Problem Solving Game. Proceedings of the 9th International Conference on Intelligent Tutoring Systems, 40-49. [pdf]

Rodrigo, M.M.T., Rebolledo-Mendez, G., Baker, R.S.J.d., du Boulay, B., Sugay, J.O., Lim, S.A.L., Espejo-Lahoz, M.B., Luckin, R. (2008) The Effects of Motivational Modeling on Affect in an Intelligent Tutoring System. Proceedings of International Conference on Computers in Education, 57-64.[pdf]

Baker, R.S.J.d., de Carvalho, A.M.J.A., Raspat, J., Aleven, V., Corbett, A.T., Koedinger, K.R. (2009) Educational Software Features that Encourage and Discourage "Gaming the System". Proceedings of the 14th International Conference on Artificial Intelligence in Education, 475-482. [pdf] [Honorable Mention for Best Paper Award]

Rodrigo, M.M.T., Baker, R.S.J.d. (2009) Coarse-Grained Detection of Student Frustration in an Introductory Programming Course. Proceedings of ICER 2009: the International Computing Education Workshop. [pdf]

Rodrigo, M.M.T., Baker, R.S., Jadud, M.C., Amarra, A.C.M., Dy, T., Espejo-Lahoz, M.B.V., Lim, S.A.L., Pascua, S.A.M.S., Sugay, J.O., Tabanao, E.S. (2009) Affective and Behavioral Predictors of Novice Programmer Achievement. Proceedings of the 14th ACM-SIGCSE Annual Conference on Innovation and Technology in Computer Science Education, 156-160. [pdf]

Baker, R.S.J.d., D'Mello, S.K., Rodrigo, M.M.T., Graesser, A.C. (2010) Better to Be Frustrated than Bored: The Incidence, Persistence, and Impact of Learners' Cognitive-Affective States during Interactions with Three Different Computer-Based Learning Environments. International Journal of Human-Computer Studies, 68 (4), 223-241. [preprint draft pdf]

Rodrigo, M.M.T., Baker, R.S.J.d., Agapito, J., Nabos, J., Repalam, M.C., Reyes, S.S. (2010) Comparing Disengaged Behavior within a Cognitive Tutor in the USA and Philippines. Proceedings of the 10th Annual Conference on Intelligent Tutoring Systems, 263-265. [pdf]

Baker, R.S.J.d., Moore, G., Wagner, A., Kalka, J., Karabinos, M., Ashe, C., Yaron, D. (2011) The Dynamics Between Student Affect and Behavior Occuring Outside of Educational Software. Proceedings of the 4th bi-annual International Conference on Affective Computing and Intelligent Interaction.[pdf]

Rodrigo, M.M.T., Baker, R.S.J.d. (2011) Comparing Learners' Affect While Using an Intelligent Tutor and an Educational Game. Research and Practice in Technology Enhanced Learning, 6 (1), 43-66. [preprint draft pdf]

San Pedro, M.O.C., Rodrigo, M.M., Baker, R.S.J.d. (2011) The Relationship between Carelessness and Affect in a Cognitive Tutor. Proceedings of the 4th bi-annual International Conference on Affective Computing and Intelligent Interaction.[pdf]

Baker, R.S.J.d., Gowda, S.M., Wixon, M., Kalka, J., Wagner, A.Z., Salvi, A., Aleven, V., Kusbit, G., Ocumpaugh, J., Rossi, L. (2012) Sensor-free automated detection of affect in a Cognitive Tutor for Algebra. Proceedings of the 5th International Conference on Educational Data Mining, 126-133.[pdf]

Hershkovitz, A., Baker, R.S.J.d., Gobert, J., Kauffman-Rogoff, Z., Wixon, M. (2012) Student Attributes, Affective States, and Engagement in Science Inquiry Microworlds. Paper presented at The European Association for Research on Learning and Instruction (EARLI) SIG 20 Conference.

Hershkovitz, A., Baker, R.S.J.d., Gobert, J., Nakama, A. (2012) A Data-driven Path Model of Student Attributes, Affect, and Engagement in a Computer-based Science Inquiry Microworld. Proceedings of the International Conference on the Learning Sciences.[pdf]

Rodrigo, M.M.T., Baker, R.S.J.d., Agapito, J., Nabos, J., Repalam, M.C., Reyes, S.S., San Pedro, M.C.Z. (2012) The Effects of an Interactive Software Agent on Student Affective Dynamics while Using an Intelligent Tutoring System. IEEE Transactions on Affective Computing, 3 (2), 224-236. [preprint draft pdf]

Pardos, Z.A., Baker, R.S.J.d., San Pedro, M.O.C.Z., Gowda, S.M., Gowda, S.M. (2013) Affective states and state tests: Investigating how affect throughout the school year predicts end of year learning outcomes. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge, 117-124.[pdf]

Baker, R.S.J.d., Ocumpaugh, J.L., Gowda, S.M., Gowda, S.M., Heffernan, N.T. (2013) Ensuring Reliability of Educational Data Mining Detectors for Diverse Populations of Learners. Presentation at CREA: Center for Culturally Responsive Evaluation and Assessment: Inaugural Conference. Published as Extended Abstract. [pdf]

Rodrigo, M.M.T., Baker, R.S.J.d., Rossi, L. (in press) Student Off-Task Behavior in Computer-Based Learning in the Philippines: Comparison to Prior Research in the USA. To appear in Teachers College Record. [preprint draft pdf]

Godwin, K.E., Almeda, M.V., Petroccia, M., Baker, R.S., Fisher, A.V. (in press) Classroom activities and off-task behavior in elementary school children. Poster paper. To appear in Proceedings of the Annual Meeting of the Cognitive Science Society.[pdf]

Hershkovitz, A., Baker, R.S.J.d., Moore, G.R., Rossi, L.M., van Velsen, M. (in press) The Interplay between Affect and Engagement in Classrooms Using AIED Software. To appear in Proceedings of the 16th International Conference on Artificial Intelligence and Education.[pdf]

Ocumpaugh, J., Baker, R.S.J.d., Gaudino, S., Labrum, M.J., Dezendorf, T. (in press) Field Observations of Engagement in Reasoning Mind. To appear in Proceedings of the 16th International Conference on Artificial Intelligence and Education.[pdf]