Undergraduate In-Class Participation in Advanced Engineering Research (UPAER)

Evaluation Report

March 1996 - Version 1.0

Vanessa Stevens

Research Associate and UPAER Project Evaluator


Joshua H. Reibel

Senior Research Associate, Assessment and Evaluation

Institute for Learning Technologies | Columbia University

Robert McClintock, Director
K. A. Taipale, Associate Director

Jump down this page to
Introduction || Evaluation Plan || Description of the Research Project || Pedagogy || Use of Technology
Findings || Conclusions and Recommendations || Acknowledgments || Instruments

Introduction (pp. 1-3)

The Undergraduate In-Class Participation in Advanced Engineering Research (UPAER) course is an innovative approach to undergraduate education being developed at Columbia University. The UPAER course combines undergraduate instruction and advanced engineering research, by involving undergraduate students in an on-going, interdisciplinary research project.

The pilot UPAER course was offered by an interdepartmental coalition of the School of Architecture, the Computer Science Department, and the Engineering School, and gave undergraduate students an opportunity to participate in building an augmented reality testbed to improve spaceframe construction procedures. The augmented reality research involves the design and development of a head-worn computer interface that provides auditory information and superposes visual information over the wearer's naturally occurring view of the world.

Students and faculty from the three departments participated in the first offering of this course, which was developed as a prototype for future course offerings at Columbia. As a prototype, it sought not only to achieve objectives within the focus of the course but also to test the effectiveness of combining instruction and advanced research and to provide for the identification of characteristics that define a promising, workable model for research-based course offerings.

The UPAER course, which was based entirely on the augmented reality research project, was organized to allow undergraduate students maximum involvement in the project. Therefore, a high degree of student participation in the design and construction of the augmented reality testbed was a priority. Project leaders hoped that such intense, real-world participation would enrich students' undergraduate academic experiences and positively influence their interest in pursuing graduate studies and careers in engineering. The course designers also hoped that the structure of the class would help impart skills of communication and collaboration necessary for effective teamwork, while concurrently advancing the research project itself.

The UPAER course was a focused effort to address the following perceived shortcomings of typical undergraduate education [note]:

In planning the UPAER course, the following goals and objectives were specified before the course began.


This evaluation of the pilot UPAER course indicates that many of the goals outlined at the beginning of the course were successfully attained. Specifically, undergraduates became integral participants in the solution of advanced research problems and, in the process, learned about the routines, techniques, and content of advanced research. Students were able to apply previously acquired theoretical knowledge to solve practical problems. Their involvement in interdisciplinary work teams strengthened their ability to communicate academic information to peers and helped them develop meaningful working relationships.

In addition to effecting positive student outcomes, the UPAER course improved the augmented reality research project and enabled the professors to realize gains of their own. The augmented reality testbed was enhanced by the participation of a diverse group of undergraduate students. The close work teams that developed around this research helped professors strengthen their own interdisciplinary collaborations.

Despite these important achievements, the group encountered some practical obstacles. The augmented reality testbed was not finished within the time frame of the fall semester, due in part to an equipment delivery delay. The difficulties encountered when coordinating research and teaching schedules illustrate some of the logistical challenges of involving undergraduate students in research projects that may have an extended life span.

Evaluation Plan (p. 4)

The Undergraduate Participation in UPAER course was evaluated by the Institute for Learning Technologies, Teachers College, Columbia University. The major focus of the evaluation was assessing the extent to which stated goals and objectives of the course were met. Attention was also paid to the progress of each of the student participants, as well as to the course's influence on the graduate teaching assistant and the professors. Informal meetings between key faculty members and the evaluator were frequent and discussions of the course methods and practice continued throughout the semester and beyond.

A variety of quantitative and qualitative instruments was used to gather baseline information, measure participants' attitudes, and assess the course's efficacy. Quantitative instruments were scored on a scale of 1 to 5 with a score of 5 indicating strong agreement with an evaluation item and a score of 1 indicating strong disagreement. [Appendices contain complete survey results, contact the Institute for Learning Technologies.] At the beginning of the course, intake surveys were distributed to each of the enrolled students to establish baseline attitudes toward engineering, research, and various aspects of undergraduate study. Throughout the semester, the evaluator attended course meetings, made observations, and occasionally shared those observations with faculty members in informal settings. At the end of the course, quantitative surveys were distributed to every person involved in the course, including students, professors, and the graduate teaching assistant. These results were recorded and assessed along with previously collected data. Interviews of every participant were also conducted within a week of the final project presentation. Products of the course, including the augmented reality testbed and the accompanying web site, were also examined.

The opportunity to conduct a comparative evaluation was limited because the content of the UPAER course did not coincide with that of other Columbia courses to such an extent that direct comparison was possible. This evaluation focused mainly on observations of the course and experiences of the participants as reported by the participants themselves. Future evaluations of UPAER courses will direct increased attention to assessing the efficacy of particular instances of teaching, learning, and research by measuring student skill development and comparing achievements in an UPAER course with those realized in other undergraduate engineering courses.

Description of the Research Project (pp. 5-10)

The UPAER course curriculum was structured entirely around the augmented reality research project. Augmented reality uses advanced technologies including see-through goggles to project additional information on the world as viewed by the user. The research project that formed the foundation for the UPAER course was the design and development of an augmented reality system that could aid in spaceframe construction. The final class requirement was the completion of a demonstration testbed augmented reality system for use in constructing a portion of a full scale space-frame. In the future, this system will be used for educational projects as well as actual on-site construction assistance. This interdisciplinary project required the input of design specialists, engineers, and computer scientists to develop the physical spaceframe demonstration system, the computer interface and the programs to control the augmented reality system, and to construct, operate and test the system.

Somewhat like virtual reality, augmented reality uses a headset to give visual and auditory information to the user. The headset, which looks like a large version of those used in 3D movie theaters, is connected to a computer that feeds information into the eyeglasses and the headphones. [See Figure 1]

Figure 1. Augmented reality goggles

Unlike virtual reality headsets, however, these goggles allow the user to see the real world. The augmented reality display projects additional information on top of that world view.

In this project, the information appearing in the goggles was generated by the computer as shown in Figure 2. The actual spaceframe base has been added to the diagram to give a point of reference.

Figure 2. Interface mock-up for strut 11.

The view from Figure 2 is then superimposed on the real world view illustrated in Figure 3 resulting in the user view shown in Figure 4.

Figure 3. Real life view of spaceframe without augmented reality.

Figure 4. Real spaceframe as seen with augmented reality interface superimposed.

Spaceframes look to a layperson like very large tinker toys, and in fact they are used in a similar fashion to construct real buildings. A spaceframe building is supported by struts that are interconnected by nodes. While elegant in structure, spaceframe buildings can be very difficult to construct because each of the nodes and struts differ greatly in function, but are similar, or may be identical, in appearance.

The concept of the augmented reality research project was to develop a system to supply a construction worker with information about each of the spaceframe pieces as needed. This information would be delivered through the augmented reality goggles, thus eliminating the need to refer to a complex manual during the construction process. A novice spaceframe construction worker could be trained or led step-by-step through a spaceframe construction operation with the guidance of the augmented reality system. [See Figure 5]

Figure 5. Worker adding a strut to the spaceframe testbed, aided by the augmented reality system.

In order to provide the computer with information about the spaceframe itself, mock-ups of the construction sequence were programmed in advance. A computer mock-up serves to locate the spaceframe and the next piece to be assembled in 3D space. In Figure 6, strut eleven appears in red, indicating its position in virtual space though it has not yet been added to the spaceframe.

Figure 6. Spaceframe mock-up with struts in white, nodes in gold, and a virtual strut in red.

For the augmented reality system to function properly, the computer must not only possess information about the spaceframe structure, but must also acquire information regarding the location of the goggles in three dimensional space. Such information enables the computer to align the projected image with the spaceframe structure. Therefore, the final design challenge was to program a device to keep track of the position of the goggles to solve this problem.

To accomplish these diverse goals, the following research objectives had to be met:

In order to acquaint each team member with the goals and objectives of the research project, the following hypothetical notion of the operation of the augmented reality testbed was proposed at the onset of the course:

The design of the augmented reality testbed for spaceframe construction was inherently interdisciplinary, requiring input from the domains of building science, computer science, engineering, and design. Starnet International, a recognized leader in spaceframe design and construction, donated a complete set of spaceframe components and showed students how spaceframe systems are currently constructed, extending the collaboration beyond the participating Columbia University departments.

When complete, the testbed will illustrate how digital technologies can be used to improve and streamline quality control, by alerting workers immediately when a construction piece is incorrectly installed. The testbed will also be used to accelerate the learning process for a variety of engineering related tasks. The augmented reality systemÕs continual on-site guidance of workers throughout the construction process will reduce, or possibly eliminate, the need for training. Both of these benefits are easily transferable beyond the domain of building construction to other disciplines.

Pedagogy (pp. 11-12)

The course was designed, both in structure and in process, to expose participating undergraduate students to a significant number of learning opportunities not normally encountered in their traditional classroom experiences. Specifically, students were given opportunities to engage in real-world problem solving that was challenging within each discipline and required interdisciplinary collaboration. Course designers posited that this high degree of intellectual and practical involvement in the research project would increase the students' acquisition of research related skills.

The goal of the course was not the execution of any particular syllabus but the completion of a working augmented reality testbed. From the first solicitations of student interest in the course, every attempt was made to insure that students understood that they would be the researchers on this team.

Students enrolled in this three credit independent study course will work as a part of a research team to help design and construct a demonstration testbed augmented reality facility. [from the course flyer]
The testbed system was to be designed, constructed, operated, and tested by a small, collaborative group of civil engineering, computer science, and architecture students. Undergraduates participated as research team members for academic credit and worked directly with the professors who have conceived this research and their graduate research assistants. Building industry manufacturers and construction companies also worked with the students to determine optimum fabrication and erection processes.

While the research project was the driving force behind the course, course designers did conceive of the undergraduate participation as occurring within the context of a class. Therefore, a rough outline of learning and research goals was assembled early on. These goals evolved during the course of the semester, but attention was paid to the importance of the project as both a research project and a structured learning experience throughout.

Course participants included three professors (two from Building Technologies/Architecture and one from Computer Science), one graduate teaching assistant (from Computer Science), and three undergraduate students (from Computer Science, Architecture, and Mechanical Engineering). Each week, the course participants met to discuss and critique the progress made since the last session and to determine the goals of the next week. These meetings were conceived as work groups, in which participants explained their modules and solicited feedback from other group members. Following weekly meetings, students pursued the completion of their tasks with each other, and with assistance and guidance from the TA and the professors. Students were charged with driving the project, insofar as they were responsible for accomplishing their tasks and subsequently conceiving the next steps necessary to advance the project.

These weekly meetings were among the overt attempts to make interdisciplinary collaboration and hands-on work the core of the class experience. The course was deliberately designed to provide students with opportunities to work on a diverse team in the interest of solving a real-world problem. Attention was paid to both the nature of the students' collaboration and the importance of its interdisciplinary quality. Additionally, two of the professors/researchers, one from building science and one from computer science, had collaborated in the past and considered this UPAER course an extension of their past research.

Use of Technology (p. 13)

The choice of building an augmented reality testbed as a topic for the course put the class on the cutting edge of technology development. Widespread use of other technologies was also instrumental to the success of this course. Because teamwork and communication were key features of the course, email was used to exchange messages and schedule additional meetings. Such constant communication was designed to help students avoid getting stuck on one issue that left them unable to complete any work before the next class meeting.

In addition, the World Wide Web was the conduit for intrateam communication and dissemination for much of the research that was developed during the UPAER course. Specifications describing the performance of the augmented reality system including 3D models of the spaceframe, audio recordings of instructions, interface mock-ups, and design parameters were published on the Web. This information was used by team members during the design and development of the project and also documented research progress.

Findings (pp. 14-48)
Conclusions, and Recommendations (pp. 49-51)

These sections contain confidential information for internal use only and are available upon request from the Institute for Learning Technologies.

For further information send email requests to info@ilt.columbia.edu or contact the Institute for Learning Technologies, Box 144, Teachers College, Columbia University, New York, NY 10027; telephone 212 678-4000; facsimile 212 678-4048.

Acknowledgments (p. 52)

Research on the augmented reality project is supported by: the Office of Naval Research under Contract N00014-94-1-0564; NSF Gateway Engineering Coalition under NSF Grant EEC-9444246; the Columbia University CAT in High Performance Computing and Communications in Healthcare, a NY State Center for Advanced Technology supported by the NY State Science and Technology Foundation; the Columbia Center for Telecommunications Research under NSF Grant ECD-88-1111; and NSF Grant CDA-92-23009; and the Columbia University Provost's Strategic Initiative Fund.

The UPAER project of the Gateway Engineering Education Coalition, (NSF Award EEC-9444246), is supported in part by the Engineering Education and Centers Division of the National Science Foundation. Starnet International Inc. provided the spaceframe. Thanks to all faculty, graduate, and undergraduate participants and undergraduate advisors: Karen Fairbanks, Director, Columbia College Architecture Program and Adjunct Assistant Professor of Architecture, GSAP and Anthony Renshaw, Lecturer, Mechanical Engineering.


For a discussion of these issues as they relate to the Engineering School at Columbia University, see Reibel, J. and Bakia, M. The Gateway Engineering Education Coalition Evaluation Report, New York: Institute for Learning Technologies, 1995. [return to text]