Grant Recipients:

Whitten, Shannon; Fiore, Stephen; Sims, Valerie

Title: Exploring the use of computational tools to grade student essays in web-based courses

Contact Information:

Psychology

Description of Grant:

The aim of this grant is to explore available software options for computational grading of essay examinations and to implement one of these tools in a course with at least 30 students, with the aim of testing the effectiveness of the tool and the impact on the student's learning.  This research could be the first step toward resolving a difficult problem for instructors of web-based courses and toward creating a distance learning environment at UCF that encourages information fluency in addition to the mastery of basic concepts.

Grant Report/Results:

Automatic Essay Grader Study

Research Questions:

• Will write differently if they are told their essays will be graded by an automatic essay grader?
• Will certain types of students write differently?

Hypotheses: Students who know their essays are being graded by a computer will write differently compared to students who believe that their essays are being graded by the instructor. Specifically,

H1.  Anaphoric References will decrease in AEG condition.

H2:  Number of Emotion words will decrease in AEG condition.

H3:  Higher proportion of clauses will tie directly to content in AEG condition.

H4:  Content that establishes common-ground will be reduced in the AEG condition, such as words or phrases about school and work.

In addition, there will be effects of student personality,

H5: Students high in Openness will write more naturally when they believe that a computer is grading their essay. 

H6: Students high in Anthropomorphic tendencies will write more naturally when they believe that a computer is grading their essay.  

Participants: 43 Undergraduates enrolled in EXP 3604, Cognitive Psychology.

Procedure: Students completed a final essay test for their online Cognitive Psychology (EXP 3604) course. Students will be randomly assigned to 1 of 2 groups:

1) Automatic-Essay Grader (AEG) condition: 21 participants were told that their essay would be graded by a computer program.

2) Instructor-Graded (IG): 22 participants were told that their essay would be graded by the Professor.

As soon as students begin the assessment, they saw the following instructions:

Write a short essay about one specific area of Cognitive Psychology (such as implicit memory, pragmatics of language or the Gestalt principles of Perception). Include in your essay how what you have learned in about this topic can apply that to something in your life: your job, your schoolwork, your future, your family, anything at all. You will be given 1 hour to complete the essay. You should be informed that this essay will be graded using an automatic computer essay grader and not by the TA or the professor. / You should be informed that this essay will be graded by the professor and not the TA for this course.

Immediately after, they will be given the NEO big five personality inventory, which is a reliable and valid measure of 5 personality traits (Costa & McRae, 1992), followed by the Anthropomorphism scale developed by Valerie Sims in the department of Psychology, followed by a demographic questionnaire.

Anaphoric references (H1) were scored using Coh-metrix (McNamara, Louwerse, Cai, & Graesser, 2005). Emotion words (H2) and words about occupations and school (H4) were calculated using Linguistic Inquiry and Word Count (LIWC, Pennebaker, Francis, & Booth, 2001). The proportion of clauses that reflected course content (H3) were scored by the professor.

Results

An independent-samples multivariate analysis of variance was performed on the data. Hypotheses 1 – 3 were not supported. However, Hypothesis 4 was supported. Students in the IG condition included more words associated with school with means of 2.03 (SD = 1.26) and 0.90 (SD = .71), respectively, F (1,41) = 13.0, p = .001.  Additionally, students in the IG condition included more words associated with occupations, with means of 3.66 (SD = 1.79) and 2.58 (SD = 1.23), respectively, F (1,41) = 5.24 p = .03.

In addition, scores on tests of individual differences did not significantly influence the results (Hypotheses 5 and 6). Further analyses revealed no differences for sentence length, etc. These data are currently being scores for other variables, such as use of examples and figurative language. In addition, we are in the process of conducting an extension of this research.

Discussion

According to the present analysis, essays written to be scored by an AEG are similar to essays written to be scored by an instructor. However, there seems to be a difference at the pragmatic level of analysis such that students are aware of their audience and write in a way that established common ground with the instructor.  Previous research has established that speakers do estimate shared knowledge before making a linguistic contribution and that they adjust their contribution on the basis of this estimation (Clark & Brennen, 1991; Fussell & Krauss, 1992; Hong; Lau, Chiu, & Hong, 2001). It is possible that such a discourse model could be applied to student essays.  The present results suggest that students are aware that they are being assessed by a computer, but this awareness does not influence the coherence, emotional expression, or substantive content of their essays. 

References

Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. In L.

B. Resnick, J. M. Levine, & S. D. Teasley (Ed.), Perspectives on

Socially Shared Cognition. Washington, D.C.: American Psychological

Association.

Fussell, S. & Krauss, R. (1992).  Coordination of knowledge in communication: Effects

of speakers’ assumptions about others’ knowledge.  Journal of Personality and Social Psychology, 62, 378 - 391.

Jones, E.A. & Nugent, M. (1996). Enhancing the Teaching and Learning Process of

Undergraduate Education at Large Universities. Paper presented at the Annual Meeting of the American Educational Research Association, New York, NY, April 8-13.

Landauer, T. K., Laham, D. & Foltz, P. (2003). Automatic essay assessment.  

Assessment in Education: Principles, Policy & Practice, 10, 295-308.

Lau, I.Y.M, Chiu, C., & Hong, Y. (2001). I know what you know: Assumptions about

others’ knowledge and their effects on message construction. Social Cognition, 19, 587 – 600. 

McNamara, D.S., Louwerse, M.M., Cai, Z., & Graesser, A. (2005). Coh-Metrix

version 1.4. Retrieved [Feb 8,2007], from http//:cohmetrix.memphis.edu.

Pennebaker, J.W. Francis, M.E. & Booth, R.J. (2001). Linguistic Inquiry and Word

Count.

Shermis, M.D., Koch, C.M., Page, E.B., & Keith, T.Z., Harrington, S. (2002). Trait

ratings for automatic essay grading. Educational and Psychological Measurement, 62, 5 – 18.

Snyder, T.D. & Dillow, S. A. (2007). Digest of Education Statistics 2006: NCES 2007

017. National Center for Education Statistics, 1- 736.

Warren, G. (1979). Essay versus multiple choice tests. Journal of Research in Science

Teaching, 16, 563-567.            

White, E. M. (1990). Assessing higher-order thinking and communication skills in

college graduates through writing. ERIC Research Report.