Open Access Research article

Order matters: pre-assessments and student generated representations

Kristy L Halverson1*, Carrie J Boyce1 and Jill D Maroo2

Author Affiliations

1 Department of Biological Sciences, University of Southern Mississippi, 118 College Dr. #5018, Hattiesburg, MS 39406, USA

2 Center for Science and Mathematics Education, University of Southern Mississippi, 118 College Dr. #5087 Hattiesburg, MS 39406, USA

For all author emails, please log on.

Evolution: Education and Outreach 2013, 6:24  doi:10.1186/1936-6434-6-24


The electronic version of this article is the complete one and can be found online at: http://www.evolution-outreach.com/content/6/1/24


Received:31 May 2013
Accepted:31 May 2013
Published:19 July 2013

© 2013 Halverson et al.; licensee Springer.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background

People have preconceived notions about how the world works from personal experiences. When these notions are scientifically inaccurate they can encourage unintended learning outcomes. Thus assessing students’ prior knowledge is important, allowing instructors to target misconceptions and optimize learning. However, the structure/administration of pre-assessments can influence students’ achievement and potentially mask errors in understanding.

Methods

We investigated the influence of pre-assessment task order on students’ tree thinking responses. We investigated student responses on a two-tiered pretest and an associated tree building task from 133 participants.

Results

Differences in the styles of student generated representations were significantly related to task order.

Conclusion

This influence creates the challenge of diagnosing student ideas and facilitating conceptual change, while not accidentally provoking misconceptions in the process.

Keywords:
Evolution; Science education; Phylogenetic representations; Pre-Assessment; Tree building

Background

Individuals have preconceived notions developed from their experiences and knowledge they have acquired from the world around them. Conclusions individuals infer from scientific concepts vary because each person’s experiences with scientific phenomena are different resulting in different conclusions depending on whether this person is or is not a scientist (e.g., Baum et al. [2005]). Everyday thinking is not always equivalent to scientifically accepted concepts. Daily experiences can influence students to draw on preconceived ideas that may result in an alternative conception (Driver et al. [1994]; Duit [1991]). For example, students’ intuitive folk taxonomy separates reptiles and birds, while grouping crocodiles with lizards and turtles based on reptilian characters. In contrast, an accurate phylogenetic grouping would place crocodiles with birds based on shared common ancestry (Figure 1). This shows tree thinking is largely inconsistent with everyday thinking about biological relatedness (Cobern et al. [1999]).

thumbnailFigure 1. A sample phylogenetic tree illustrating evolutionary relationships among four taxa.

According to Novick and Catley ([2008]), substantial current college biology curriculum is driven by the use of phylogenetic trees. However, it is clear that most students do not interpret trees in the same manner as evolutionary biologists (Baum et al. [2005]; Gregory [2008]; Halverson et al. [2011]; Meir et al. [2007]). Students’ alternative conceptions, such as folk taxonomy groupings, can cause individuals to generate a variety of inaccurate responses when interpreting and explaining scientific concepts. If a person’s initial knowledge is inaccurate, when confronted with new, scientifically accurate explanations, they may not accept the new idea because it does not fit into their prior understanding (Bransford et al. [2000]; Chi and Roscoe [2002]; Duit & Treagust [2003]; Posner et al. [1982]).

Current quantitative assessment instruments (e.g., Baum et al. [2005]) may uncover categories of student errors, such as those described by Gregory ([2008]), but they do not uncover the underlying reasoning patterns. Thus, assessing and diagnosing students’ prior knowledge is important in classroom settings to allow teachers to target alternative conceptions and optimize learning. However, the structure or even the administration of a pretest itself can influence students’ achievement and potentially mask errors in understanding (Solomon [1949]). The purpose of this study was to investigate how student generated representations changed based on when they were given a tree building task during a pre-assessment (e.g., before or after tree thinking questions).

Theoretical framework

Learning can be described as a progression of change from the learner’s prior naïve conceptions to intermediate conceptions to scientific conceptions (Niedderer and Goldberg [1994]). In this process, the individual constructs personal meaning by interpreting new information through a lens of prior knowledge; in other words, new knowledge builds upon prior knowledge (Bransford et al. [2000]). Many educators use pretests as a method of assessing students’ prior knowledge and diagnosing alternative conceptions in a particular content area (Berry [2008]). It has been suggested that taking a pretest can influence students’ perceptions of content, influencing student performance on the assessment (Solomon [1949]). This can be explained as a form of intuitive heuristics. Maeyer and Talanquer ([2010]) describe intuitive heuristics as a process students use to answer questions through simplification of their reasoning process. Students use less information than scientists to determine scientific relationships and use a series of potentially arbitrary rules when deciding where to look for information and how to answer questions based on the information gathered (Maeyer and Talanquer [2010]). Students may answer assessment questions using short-cuts: (1) students’ recognition of the material; (2) the similarities they see between subject material (e.g., similarities between taxa); and (3) external cues that promote choosing one answer over another (e.g., one answer is more in line with everyday thinking than accurate scientific thought or reading the trees across the tips) (Maeyer and Talanquer [2010]). Students may consciously or unconsciously use these rules or short-cuts when presented with tree building or tree reading assessments.

Tree building (generating a phylogenetic diagram) is a cognitively more difficult task than tree reading (interpreting and comparing phylogenetic diagrams) and builds upon a student’s tree reading ability (Halverson [2011]). One common problem individuals have is incorrectly reading phylogenetic trees across the tips, looking only at extant taxa rather than correctly interpreting the nodes (Crisp and Cook [2005]). Maeyer and Talanquer ([2010]) found a similar problem in chemistry where students used arbitrary trends in the periodic table to answer questions when comparing individual atoms.

Researchers have used a two-tiered pretest to assess student tree thinking by presenting multiple styles of phylogenetic trees for students to interpret and compare (see Halverson et al. [2011]). A traditional two-tiered pretest consists of two parts: the first is a multiple-choice, the second requests the students explain their answer choice. On this pretest students were also asked to generate a visual representation illustrating the relationships among 14 plant, fungi, and animal taxa and to explain the relationships drawn. While we accept that task order can influence students’ learning, it is unknown if task order on a single pretest influences students’ responses on the given assessment.

Research questions

In order to identify how student generated representations changed based on when they were given a tree building task during a pre-assessment, we asked the following three research questions: 1) What styles of representations did students generate when given a tree building task?; 2) How do student justifications of representations differ when students are given a tree building task before versus after a tree reading assessment?; and 3) What is the connection between student generated representations and student tree reading?

Methods

Participants came from two public research universities across four semesters, for a total of 133 students. Students were enrolled in one of two upper-level biology courses. Both courses used phylogenetic trees heavily during instruction, highly emphasizing tree thinking. Each course ranged from about 30 to 50 students and was designed for life sciences majors. There was no significant difference between the sample populations at each university for the last two semesters (t(73) = .608, p = .545). Thus, we considered the two populations as one for the purpose of this study. We gave students enrolled in the first two semesters a tree thinking pretest. The last question of this pretest was a tree building task. We changed the order of pretest questions for students enrolled in the last two semesters. For these students the tree building task was the first question on the pretest.

Data sources

Data came from student responses on a two-tiered pretest and associated tree building task (see Halverson et al. [2011]). The pretest assessed students’ tree reading skills. The tree building task asked students to draw a visual representation that illustrated their understanding of how given taxa were related to one another. Students also provided written descriptions explaining their representation.

Data analysis

For the qualitative portion of our data analysis, we used a deductive method to code the types of representations students generated and students’ justifications for these images, on the pretest. We identified four categories of student generated representations: accurate tree representations, alternative tree representations, alternative tree-like representations, and alternative non-tree-like representations (see Halverson [2011]). We further identified the nature of the student generated representations as appearing tree-like in nature (accurate tree representations, alternative tree representations, and alternative tree-like representations) or non-tree-like in nature (alternative non-tree-like representations). To improve the reliability of our findings, each researcher coded all student responses. We compared codes and had 100% inter-rater reliability.

As a secondary analysis, we grouped student representations into categories of diagrams according to the justifications students provided for their images. Phylogenetic diagrams were the only representations for which students used scientific explanations to justify accurate evolutionary relationships among taxa. Students justified relationships in ecological diagrams by relying on factors such as habitat, trophic level, etc. Students explained that they viewed evolution as progressive or that organisms evolved into one another when they generated diagrams representing alternative evolutionary principles.

For the quantitative portion of our data analysis, we scored the accuracy of students’ tree thinking at the beginning of the semester by giving students one point for selecting the appropriate multiple choice response and one point for providing an accurate explanation for why they selected their answer (maximum of two points per question). We then assigned numeric values to the style of representations students generated on the tree building task. In order to determine the relationship between task order and its influence on student generated phylogenetic trees, we conducted a Pearson’s Chi-square test. For this analysis we grouped student generated representations into four categories: phylogenetic diagrams, ecological diagrams, diagrams representing alternative evolutionary principles, or no representation. We compared these styles of representations with respect to when the task was administered, before or after the pretest. We used Yates’s continuity correction in place of χ2 to account for the expected cell count violation. Statistical significance was assigned when p < 0.05. We also analyzed the likelihood of students generating tree-like diagrams in relation to task order by calculating tree-like and non-tree-like frequency counts.

We analyzed the relationship between student tree thinking and their generated representations by running an independent t-test to determine the statistical significance between assessment scores and the tree-like nature of the representations generated. For this analysis we grouped the style of representations students generated into two categories: tree-like and non-tree-like. To further explore the population we divided the population into four groups and ran a one-way ANOVA (Figure 2). Three planned contrasts were decided a priori comparing four groups: students who were given the tree building task after the pretest and drew a tree-like representation, students who were given the tree building task after the pretest and drew a non-tree-like representation, students who were given the tree building task before the pretest and drew a tree-like representation, and students who were given the tree building task before the pretest and drew a non-tree-like representation.

thumbnailFigure 2. Three planned contrasts of the ANOVA design.

Results & discussion

We organized our findings into three major sections. First, we describe the frequency and types of representations students generated. Second, we describe the representation style differences related to tree building to task order. Finally, we describe the relationship between the representation styles generated and students’ tree reading.

Styles of representations generated

Generating a phylogenetic tree involves isolating, interpreting, and using data as evidence of evolutionary relationships. There are many different styles of representation an expert could generate if asked to draw a visual representation illustrating the relationships among the following organisms: bat, onion, dolphin, parrot, oak tree, fern, pine tree, slug, daisy, human, trout, algae, mushroom, turtle, crab, and fly. Of the varying representation styles, scientifically accurate representations share the following features: relationships are grouped based on evolutionary histories and common ancestry, all organisms are considered to be related and are connected within a single representation, taxa are placed at the terminal tips assuming there are hypothetical ancestors at the nodes, and consensus nodes are used when relationships are unknown and result in the tree being non-bifurcating at that location. This can cause the tree to have three or more branches coming off a single node.

We asked students to consider the same 16 organisms listed above and draw a visual representation that illustrated their understanding of how the organisms are related to one another. We categorized student generated representations and found that only eight participants generated a scientifically accurate style of representation (Table 1). The remainder of students generated alternative representations or did not attempt a representation.

Table 1. Styles of student generated representations

We further hypothesized that exposing students to phylogenetic trees on the pretest before giving them the tree building task would influence the likelihood that students would create more tree-like diagrams (accurate tree representations, alternative tree representations, and alternative tree-like representations) rather than non-tree-like diagrams. We found that the students who completed the pretest before completing the tree building task were nearly three times more likely to create tree-like representations than students who completed the tree building task at the beginning of the pretest (Table 2).

Table 2. Changes in physical representation styles in relation to task order

Student justification differences

We grouped participants’ representations into four categories according to the biological explanations students provided justifying their images: phylogenetic diagrams, ecological diagrams, alternative evolutionary principles, or no representation. Overall, the majority of students (88%) generated an alternative style of representation on the pretest. We found a statistically significant difference in the styles of representations students generated in relation to task order (χ2(3) = 22.27, p = < .001). When students were given the tree building task after the pretest they were more likely to generate trees grounded in an evolutionary basis: more than half (58%) of the students generated diagrams representing alternative evolutionary principles and 11% of students generated scientifically accurate phylogenetic diagrams. When the tree building task was given before the pretest, students were more likely to generate trees grounded in an ecological basis: the majority of students (62%) generated ecological diagrams (Table 3). This suggests that task order influences the biological basis students used to determine what it means for taxa to be related to one another.

Table 3. Change in content of representation styles in relation to when tree building task was given

Student representations and tree reading

We compared the styles of students’ representations to their tree reading pretest scores using an independent group t-test and found there was a statistically significant relationship (t(131) = 2.191, p = .03) between the tree-like nature of the representation styles and students’ tree reading scores. Although we found that Levene’s Test for Equality of Variance was not significant, the significant results of the t-test may be a factor of the category size differential rather than differences in student scores. This unknown is due to the non-tree-like category being 45% larger than the tree-like category.

To explore this relationship further we ran a one-way ANOVA with three planned contrasts and found a statistically significant difference in our four group means (F(3,129) = 3.12, p = .028) (Table 4). We ran this ANOVA using the square root of the pretest score rather than the pretest scores to correct for our violation of Levene’s Test for Equality of Variance. Planned Contrasts 1 and 3 found no significance between testing groups (Table 5). Planned Contrast 2 found a statistically significant difference between testing groups (t(129) = −2.6, p = .01). Specifically, we found that students with higher tree thinking scores were more likely (t(129) = 2.6, p = .01) to draw tree-like representations when given the tree building task after the pretest.

Table 4. ANOVA table

Table 5. Planned contrast table

Conclusions

Preassessments are often used to evaluate students’ prior knowledge at the beginning of a course (Berry [2008]). This study provides evidence that assessments used to diagnose students’ prior ideas can actually present information that influences their understanding. In some cases of this study, the assessments presented new information to some students as they had no prior experience or no recollection of seeing phylogenetic trees. Maeyer and Talanquer ([2010]) found that errors and bias may arise because students are looking at the face value of an assessment. By looking only at the surface of the assessment rather than attempting to understand the content, students are not making a connection between the tree reading questions and the tree building task the way the expert biologists would. Students may use shortcut strategies to answer assessment questions regarding the biological relatedness of organisms. The students who took the assessment with the tree building task as the last question may have been influenced by the tree reading questions and used the shapes found in those questions to pattern the physical shape of the representations they generated. Additionally, the arbitrary trend of reading a phylogenetic tree based on the relative position of the organisms across the tips of the trees (Halverson et al. [2011]) is reflected in how students justify the representations they generated. Thus, students with higher tree thinking ability are more likely to generate tree-like representations when given the tree building task after the pretest than those students given the tree building task before the pretest.

Using a tree reading pretest and a tree building task in tandem can be used to gain an understanding of students’ prior knowledge (Halverson et al. [2011]). However, we found that task order influenced student responses on a single tree building task. The connection between task order and pretest scores indicates that using an evaluative activity as a pretest question can serve as a prior knowledge diagnostic tool for courses that explore evolutionary relationships. Exposure to the phylogenetic diagrams helps students think about biological relatedness more in terms of evolution, even if inaccurate, rather than in ecological terms which is more consistent with everyday experiences (Staub et al. [2006]). However, simply exposing students to phylogenetic diagrams will not increase their tree thinking skills; this can only be accomplished through explicit instruction.

The influence of task order can create a challenge for diagnosing student ideas. Educators must remember that task order on preassessments can influence student responses and plan assessments with this knowledge in mind. While these diagnostic tools can inform future instruction, they may mask additional alternative understandings. By expanding this study to include additional samples from more universities, educators will develop a deeper understanding of the relationship between task order and how students represent their ideas.

Our study provides evidence that assessments used to diagnose students’ prior ideas can present new information that influences their current understanding. We found that task order influenced student responses on a single tree building task. Exposure to the phylogenetic diagrams helped students think about biological relatedness more in terms of evolution, even if inaccurate, rather than in ecological terms as is more consistent with everyday experiences.

Competing interest

The authors declare that they have no competing interests.

Authors’ contributions

KLH was in charge of data collection. CJB and JDM were in charge of data analysis. All authors read and approved the final manuscript.

References

  • Baum, DA, Smith, SD, Donovan, SSS (2005). The tree-thinking challenge. Science, 310, 979–980. PubMed Abstract | Publisher Full Text OpenURL

  • Berry, T (2008). Pre-test assessment. American Journal of Business Education, 1, 19–22. OpenURL

  • Bransford, JD, Brown, AL, Cocking, RR (2000). How people learn: brain, mind, experience, and school. Washington D.C: National Academy Press. Expanded ed.. OpenURL

  • Chi, MTH, & Roscoe, RD In Limon M, Mason L (Eds.) (2002). The process and challenges of conceptual change. Reconsidering conceptual change: issues in theory and practice (pp. 3–27). Dordrecht: Kluwer. OpenURL

  • Crisp, MD, & Cook, LG (2005). Do early branching lineages signify ancestral traits? Trends in Ecology & Evolution, 20, 122–128. PubMed Abstract | Publisher Full Text OpenURL

  • Cobern, WW, Gibson, AT, Underwood, SA (1999). Conceptualizations of nature: an interpretive study of 16 ninth graders’ everyday thinking. Journal of Research in Science Teaching, 36, 541–564. Publisher Full Text OpenURL

  • Driver, R, Squires, A, Rushworth, P, Woods-Robinson, V (1994). Making sense of secondary science: research into children’s ideas. London: Routledge. PubMed Abstract | Publisher Full Text OpenURL

  • Duit, R In Glynn SM, Yearny RH, Britton BK (Eds.) (1991). Students’ conceptual frameworks: consequences for learning science. The psychology of learning science Earlbaum: Mahwah, NJ. OpenURL

  • Duit, R, & Treagust, DF (2003). Conceptual change: a powerful framework for improving science teaching and learning. International Journal of Science Education, 25, 671–688. Publisher Full Text OpenURL

  • Gregory, TR (2008). Understanding evolutionary trees. Evolution: Education and Outreach, 1, 121–137. Publisher Full Text OpenURL

  • Halverson, KL (2011). Improving tree-thinking one learnable skill at a time. Evolution: Education and Outreach, 4, 95–106. Publisher Full Text OpenURL

  • Halverson, KL, Pires, JC, Abell, SK (2011). Exploring the complexity of tree thinking expertise in an undergraduate plant systematics course. Science Education, 95, 794–823. Publisher Full Text OpenURL

  • Maeyer, J, & Talanquer, V (2010). The role of intuitive heuristics in students’ thinking: ranking chemical substances. Science Education, 94, 963–984. Publisher Full Text OpenURL

  • Meir, E, Perry, J, Herron, JC, Kingsolver, J (2007). College students’ misconceptions about evolutionary trees. American Biology Teacher, 69, 71–76. Publisher Full Text OpenURL

  • Niedderer, H, & Goldberg, F (1994). An individual student’s learning process in electric circuits (Paper presented at the NARST Annual Meeting). CA: Anaheim. PubMed Abstract OpenURL

  • Novick, LR, & Catley, KM (2008). Assessing students’ understanding of cladograms (Paper presented at the National Association for Research in Science Teaching Annual Meeting). MD: Baltimore. PubMed Abstract OpenURL

  • Posner, GL, Strike, KA, Hewson, PW, Gertzog, WA (1982). Accommodation of scientific conception: toward a theory of conceptual change. Science Education, 66, 211–227. Publisher Full Text OpenURL

  • Solomon, R (1949). An extension of control group design. Psychological Bulletin, 46, 137–150. PubMed Abstract OpenURL

  • Staub, NL, Pauw, PG, Pauw, D (2006). Seeing the forest through the trees: helping students appreciate life’s diversity by building the tree of life. The American Biology Teacher, 68, 149–151. Publisher Full Text OpenURL