Research and Evaluation

At the completion of this section, you should be able to:

• define and describe the purpose of formative evaluation.
• conduct an expert evaluation of instructional materials.
• conduct a small group evaluation of instructional materials.
• conduct a field trials of instructional materials.
• apply evaluation data to the revision of instructional materials.
• define and describe the purpose of summative evaluation.
• conduct expert judgments and field trails to summative evaluation.
• define and describe the purpose of evidence-based programs.
• apply evidence-based practices to an instructional setting.

Begin by viewing the class presentation in Vimeo. Then, read each of the sections of this page.

Explore each of the following topics on this page:

• Formative Evaluation and Revision
  
• Summative Evaluation
• Evidence-based Programs and Practices
• Resources

Formative Evaluation and Revision

Formative evaluation is the "collection of data and information during the development of instruction that can be used to improve the effectiveness of the instruction" (Dick, Carey, & Carey, 2011, 257).

The purpose of formative evaluation is to identify specific errors and problems that can be corrected. They suggest focusing on the effectiveness of the instructional strategies by focusing on questions addressed to experts, instructors, and students (Dick, Carey, & Carey, 2011, 259):

• Instructional Expert
  • Are the materials appropriate for the type of learning outcome?
• Content Area Expert
  • Do the materials include adequate instruction on the subordinate skills, and are these skills sequenced and clustered logically?
• Target Audience Member
  • Are the materials clear and readily understood by representative members of the target group?
  • What is the motivational value of the materials? Do learners find the materials relevant to their needs and interests? Are they confidence are they work through the materials? Are they satisfied with what they have learned?
• Instructor
  • Can the materials be managed efficiently in the manner they are mediated?

Data can be collected using checklists, interviews, tests, surveys, and observations.

Formative evaluation involves four activities.

Expert Evaluation

Involve subject-matter experts (SME), learning specialists, and instructors in examining the instructional materials.

Once instructional designers have completed their first draft of the materials, they are too close to the process to identify problems. Even simple grammatical errors are often overlooked.

Experts can be brought in to examine the materials for accuracy and currency. In addition, instructors can point out practice, "real-world" type problems. The designer will want to immediately make changes related to error in content. However, they may wait to make other changes until after seeing the impact of the materials on students.

One-to-One Evaluation

Although known as clinical evaluation, one-to-one involves working with individual learners. Data collecting from a number of individual is used for revision. Dick, Carey, and Carey (2011, 262) suggest three criteria be used during this phase:

• Clarity. Is the message, or what is being presented, clear to individual target learners?
  • Clarity of Message. Vocabulary, sentence and message complexity, instruction, elaboration, conclusions, transitions
  • Clarity of Links. Contexts, examples, analogies, illustrations, demonstrations, reviews, summaries
  • Clarity of Procedures. Sequence, segment size, transition, pace, variation
• Impact. What is the impact of the instruction on individual learner's attitudes and achievement of the objectives and goals?
  • Impact on Attitudes. Utility of the information and skills, how easy/difficult the information and skills are to learn, satisfaction with skills learned
  • Impact on Achievement. Clarity of directions and items for posttest, scores on posttest
• Feasibility. How feasible is the instruction given the available resources (time/context)?
  • Feasibility of Learners. Maturity, independence, motivation
  • Feasibility of Resources. Time, equipment, environment

When selecting learners for the one-to-one evaluation, look for students that represent the low, middle, and high ability categories. You want to see how the range of learners will react to the materials.

Learners can record their thoughts in whatever format works best. They may highlight areas on paper, use post-it notes, or record audio as they move through the materials. The observer may ask questions as they move through the materials. It's also useful to video record the experience. It's essential that the learner is relaxed and realized that their role is to help the designer critique the materials. When problems are encountered, the learner needs to be reassured that the problem is with the materials, not the learner.

After all data has been collected, the designer will make changes in the instructional materials. However it's important not to overgeneralize from the small population. Consider individual differences before making changes.

Small Group Evaluation

In small group evaluation, eight to twenty people representing the target audience are studied. This phase is intended to help the designer determine the effectiveness of the changes made after the one-to-one evaluation and see what happens when there's no "guide on the side".

Like the one-to-one evaluation, it's important to pick a heterogeneous group that represents the target audience.

Dick, Carey, and Carey (2011, 267-268) recommend the following questions following the instruction:

• Was the instruction interesting?
• Did you understand what you were supposed to learn?
• Were the materials directly related to the objective?
• Were sufficient practice exercises included?
• Were the practice exercises relevant?
• Did the tests really measure your knowledge of the objectives?
• Did you receive sufficient feedback on your practice exercises?
• Did you feel confident when answering questions on the test?

Again, modifications to the instructional materials take place based on the results of the small group evaluation.

Field Trial

During the field trial, the materials are tested in a "real-world situation".

The purpose of the field trial is to ensure that all of the problems from earlier evaluations have been corrected and to see whether the instruction works in the intended context.

The instructional materials should be tested in a "real world" situation with a typical group of learners. The procedures followed should be those outlined in the instructional materials.

Finally, any final revisions can be made.

Revisions

Revision of materials can be an ongoing process. In areas such as technical instruction, accuracy and currency is an ongoing problem.

Evaluation forms can be used to gather information from students about the value of a course or workshops. They may be asked about specific strengths and weaknesses as well as areas they would change. Increasingly evaluation forms are being offered by on paper as well as through a phone or computer.

Below are examples of generic evaluation forms:

• Course and Faculty Evaluation Form from Idaho State University

 

Summative Evaluation

Summative evaluation is "the process of collecting data and information in order to make decisions about the acquisition or continued use of some instruction" (Dick, Carey, & Carey, 2011, 319).

While formative evaluation is used to improve the effectiveness of instruction, the purpose of summative evaluation is to verify the effectiveness of instruction with the target audience and decide whether the materials should be used in the future.

Summative evaluation generally involves two phases.

First, an expert is used to judge whether the instructional meets institutional needs. Dick, Carey, and Carey (2011, 321) list five activities:

1. evaluating the congruence between the organization's instructional needs and candidate instruction,
2. evaluating the completeness and accuracy of candidate instruction,
3. evaluating the instructional strategy contained in the candidate instruction,
4. evaluating the utility of the instruction, and
5. determining the user's satisfaction with the instruction.

Second, a field trial is used to determine the effectiveness of the instructional with the target audience. Dick, Carey, and Carey (2011, 321) identified two components:

1. Outcomes Analysis - determining the effect of the instructional on learners' skill, on the job, and on the organization
2. Management Analysis - assessing instructor and supervisor attitudes related to the learner performance, implementation feasibility, and costs

Read!
Read Strittmatter, Connie & Bratton, Virginia (September 2014). Plagiarism Awareness among Students: Assessing Integration of Ethics Theory into Library Instruction. Think about the approaches and tools these researchers used to check the effectiveness of their instruction.

 

Evidence-based Programs and Practices

We know that library programs make a difference in learning. Your role is to develop an effective, inquiry-based learning program. This program must focus on working collaboratively with others and documenting your work and the work of learners.

Scientifically-Based Research

From personal experiences and intuition to observation and research, librarians use a wide variety of techniques to make instructional decisions. With increasing pressures for accountability, school reform, and standards-based curriculum, professional librarians must refine their teaching practice to reflect research-based knowledge. This begins with applying scientifically-based research.

According the No Child Left Behind Act (2001), scientifically-based research is defined as

"research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs."

This research must include systematic, empirical methods; rigorous data analyses, reply on measurements or observational methods; use experimental or quasi-experimental designs; ensure replication; and be accepted by independent sources.

An emphasis on evidence-based education policy led the National Research Council to study the nature of scientific inquiry in education. The report, Scientific Research in Education (2002) focuses on methods for scientifically-based educational research and the appropriate use of these methods. The foundations of scientific inquiry are the same across fields focusing on the “continual process of rigorous reasoning supported by a dynamic interplay among methods, theories, and findings” (p. 2). The study also noted that “rarely does one study produce an unequivocal and durable result; multiple methods, applied over time and tied to evidentiary standards, are essential to establishing a base of scientific knowledge” (p. 2).

Scientific Research in Education (2002, 3-5) also states that a healthy community of researchers should work collaborative and follow six guiding principles of scientific inquiry:

1. pose significant questions that can be investigated empirically
2. link research to relevant theory
3. use methods that permit direct investigation of the question
4. provide a coherent and explicit chain of reasoning
5. replicate and generalize across studies
6. disclose research to encourage professional scrutiny and critique).

The State Educational Technology Director’s Association (2003) has developed a comprehensive website to disseminate information related to scientifically-based research approaches in educational technology. They suggest an eight step process for designing a scientifically-based research study including creating a plan, generating questions, establishing a conceptual framework, designing methodology, acquiring resources, implementing project, synthesizing findings, and reporting findings.

Read!
Read Making Online Instruction Count: Statistical Reporting of Web-Based Library Instruction Activities by Tim Bottorff and Andrew Todd (2012).
Think about the importance of collecting data in all instructional situations.

 

Evidence-based Libraries

In 2003, the ALA Resolution: School Libraries and Librarians are Critical to Educational Success states that numerous research studies conducted in the states of Alaska, Colorado, Iowa, Kentucky, Massachusetts, Michigan, New Mexico, Oregon, Pennsylvania, and Texas have shown a high correlation between exemplary school library programs led by a certified school librarian and student achievement on state standardized tests.

With an increasing focus on accountability, it's more important than ever for librarians of all types to document their work with faculty and students. Evidence-based programs carefully document and study how the library program makes a difference in learning. Their programs are constantly revised to reflect their findings.

In Our Instruction DOES Matter! Data Collected From Students' Works Cited Speaks Volumes, Poinier and Alevy (2010), discuss their development and application of an assessment for library instruction applied to a class assignment in conjunction with high school classes. The class that received no library instruction used unreliable web resources in their projects. However the class that received library instruction used authoritative works that were cited. They concluded that "collaborative planning and instruction by the teacher-librarian leads to both an increase in the use of reliable resources and an improved works cited product as compared to a class that has simply 'used the library'."

Read!
Read Adams, Nancy E. (2014). A Comparison of Evidence-Based Practice and the ACRL Information Literacy Standards: Implications for Information Literacy Practice.

 

Action Research and Library Media Programs

Action research is an approach used by educators to evaluate teaching methods and learning processes. It is "process of systematically evaluating the consequences of educational decisions and adjusting practice to maximize effectiveness” (McLean 1995).

Through action research, educators gain insights and improve their teaching practice. Action provides an orientation to research, a form of professional practice, a research process, and a reflective way of teaching (Arhar, Holly, & Kasten, 2001).

Action research has a long history. Although the specific origins of action research are open for debate, many have placed it with the work of Kurt Lewin (1946). He viewed action research as a tool for generating knowledge about a social system while at the same time attempting to change it. In 1953, Steven Cory stated that action research is conducted in authentic classrooms to solve practical problem using scientific methods to determine, on the basis of objective evidence, whether or not goals have been accomplished (Herrick, 1992). Action research can assist educators with a systematic method for determining what is best practice and policy for their students and can promote educational reform (Heckman, 1996; Zuber-Skerritt, 1991).

Accountability

Educators have always been accountable for their actions. However, with the No Child Left Behind Act, accountability has become a focus of schools (Schultz, 2002, p. 38). Accountability refers to the connection between student performance and the teacher’s preparedness. Through action research, educators use sound data to evaluate their educational programs. Action research is used to examine and address a particular problem, which ultimately leads to an improvement in teaching practices and assessment (Herrick, 1992, p. 47). Action research is directly linked to the legislation (NCLB). Teachers are able to show that they are making a difference with their teaching for all students and it allows them to study the effects of their teaching on student learning (Mills, 2003).

Continual Professional Development

According to Calhoun (2002), action research focuses on continual professional development—a direct route to improving teaching and learning. According to Calhoun, action research asks educators to study their practice and its content, explore research base for ideas, compare current practices with what they find, participate in training to make need changes, and look at the effects on the students and themselves (p. 18). Action research is a sound tool to help educators solve problems or investigate how students learn and succeed. Action research is a powerful tool used by teachers and administrators to measure the schools effectiveness.

Real Problems

Action research involves educators on a personal level with real problems that need solved and provides educators the opportunity to reflect on and act on real-life problems encountered in their own practices.
In the case of the proposed project, action research involves collaboration at all levels including students, researchers, and practitioners.

The research and the practice inform each other to help develop new knowledge and effective action (Ziegler, 2001). Action research is different from other research. It focuses on problems that happen in everyday life and in the learning process. Action research uses quantitative and qualitative methods. It uses qualitative to describe what is happening to understand the learning processes. Another difference is that action research is used to take action to make the necessary changes in teaching methods to help all students succeed.

In the article, A Bumpy Road to Action Research, the link needed to professionalize teaching and teachers is action research (Gilbert & Smith, 2003). Many educators and research feel strongly that action research should be a mandatory component of teacher education (Gilbert & Smith, 2003).

As self-directed professional development, action research asks the educator to start with their own knowledge, experiences, and perspectives and use these to improve teaching methods. It is unique research used to reach a specific goal. Gilbert and Smith (2003) conducted a study on the use of action research in schools. They found that when teachers view good teaching as good research, they open the gates to the future. Teachers will see how the system can be changed through their research, creating new knowledge that empowers them and their learner. Teachers have the goal of helping students become life-longer learners. Shouldn’t teachers be life-long learners, too? Another result of the their study, showed how teachers appreciated sharing their action research with each other. The teachers found it very valuable to see what others have done and changes they have made.

Action Research Takes Time

Although action research appears to be an ideal method to evaluate schools’ effectiveness and improve learner outcomes, it is not used thoroughly in schools because teachers lack the time and expertise to engage in action research. In a study of action research by teachers, Gamoran (2003) found that educators need time to develop their skills and conduct research. They also need the support of administration. When these two elements are in place, teachers were able to collaborate and collectively focus on improving student learning.

Action Research Changes Thinking

Action research changes how educators think about themselves, their work, and their school. Teachers become committed to action research because it gives them tangible results in their work and in students’ performance (Senese, 2002). Action research is used to generate data to measure the effects of various teaching methods, help make effective decisions, and assist teachers in learning new practices (Calhoun, 2002).

Hoshmand and Polkinghorne (1992) noted that the traditional concept of the relationship between science and practice has generally posited a one-way influence of science on practice. This perspective is limiting and underutilizes and undervalues the ways of knowing that are germane to practice. Action research brings those two realms of knowledge and experience together. Viewed as a frame of mind, action research calls us to a continued interest in serving our students better and providing increased accountability for our teaching. As such, action research is not simply a good idea. Instead, it becomes an ethical responsibility for monitoring the effectiveness of our practice and increasing the competency of our teaching (Elliott, 1991).

Applying Evidence to Professional Practice

Evidence-based programs use the data they collect from their own program, as well as applying scientifically -based evidence from other programs.

Many quality research studies have been conducted on topics of interest to school library media specialists on topics related to information inquiry. As you explore research studies, consider how the results can be duplicated or adapted for your program. Think about ways that you could apply the results of one of these articles to your professional practice.

Meta Analysis

Rather than wading through volumes of research studies, some educators prefer to focus on quality meta- analysis research.

According to Bangert-Drowns and Rudner (1991), “meta-analysis is a collection of systematic techniques for resolving apparent contradictions in research findings. Meta-analysts translate results from different studies to a common metric and statistically explore relations between study characteristics and findings”.

Meta-analysis has become an increasingly common and widely supported means of synthesizing findings from research (Glass 2000). According to Viadero (2002), this type of systematic review of research provides empirical evidence that certain approaches are effective in improving student learning.

Try It!
Choose ONE of the following research articles. Think about ways that you could apply the results of one of these articles to your professional practice. Some of these are older articles. How could the ideas be adapted for today's libraries.

Seeing Different: Portrayals of Disability in Young Adult Graphic Novels by Marilyn Irwin and Robin Moeller (SLMR, 13, 2010). This study explores the representation of disabilities in graphic novels. How might this impact your use of graphic novels?

Challenges to Teaching Evaluation of Online Information: A View from LM_NET by Frances Jacobson Harris (SLMR, 12, 2009). This article explores the issues in teaching evaluation skills. 

Mental Models of Information: The 1993-94 AASL/Highsmith Research Award Study by Judy M. Pitts (SLMR, 23(3), 1995). The purpose of this study was to examine the question: When students are seeking and using information, why do they make the decisions they make? 

Current Research: A Study of High School Students' Online Catalog Searching Behavior by Shu-Hsien Chen (SLMR, 22(1), 1993). This study investigated the search behavior of high school students using an online catalog. 

The Science Library Catalog: A Springboard for Information Literacy by Virginia A. Walter, Christine L. Borgman, and Sandra G. Hirsh (SLMQ, 24(2), 1996). This study explores student use of catalogs.

Information Search Process: A Summary of Research and Implications for School Library Media Programs by Shu-Hsien Chen (SLMQ, 18(1), 1989). The author summarizes a series of five studies on students' perspective of information seeking in response to a research assignment.

The Effect of High-Quality Instruction on Reading Outcomes (ASCD, 3(3), 2005). Explore reading achievement of students in high school English.

Resources

Arhar, J., Holly, M., & Kasten, W. (2001). Action research for teachers: Traveling the yellow brick road. Prentice Hall, Inc.

Bangert-Drowns, Robert L. & Rudner, Lawrence M. (1991). Meta-analysis in educational research. Practical Assessment, Research & Evaluation, 2(8).

Calhoun, E. (2002, March). Action research for school. Educational Leadership, 59(6), 18-24.

Carnahan, Danielle and Fitzpatrick, Michele. (Spring 2003) Digging out: how to avoid getting buried under a mountain of research. NCREL’s Learning Point, 8-11.

Dick, Walt, Carey, Lou, and Carey, James O. (2011). The Systematic Design of Instruction. Seventh Edition. Pearson.

Elliott, J. (1991). Action research for educational change. Open University Press.

Gamoran, A. (2003). What are they thinking? National Staff Development Council, 24 (2), 56-60.

Gilbert, S. & Smith, L. (2003). A bumpy road to action research. Kappa Delta Pi Record, 39 (2), 80-83.

Heckman, P. E, (1996). The courage to change: Stories from successful school reform. Corwin Press.

Hoshmand, L. T., & Polkinghorne, D.E. (1992). Defining the science-practice relationship and professional training. American Psychologist, 47 (1), 55-66.

Lewin, K. (1946). Action research and minority problems. Journal of Social Issues, 2(1), 65.

Loertscher, David V. (March 2003). School Library Media Programs and Academic Achievement: A Bibliography and Availability List.

McLean, J.E. (1995). Improving education through action research : A guide for administrators and teachers. Corwin Press.

Mills, G. (2003). Action research. Pearson Education, Inc.

Poggi, Sheryl. (Spring 2003) Wake-up Call: Facing the challenge to use scientifically based research in schools. NCREL’s Learning Point, 4-7.

Poinier, Sara & Alevy, Jennifer (February 2010). Our instruction DOES matter! Data collected from students' work cited speaks volumes. Teacher Librarian, 37(3), 38-39.

Schultz, R. (2002). Teachers as learners. Gifted Child Today, 25 (4), 38-45.

Senese, J. (2002). Energize with action research. National Staff Development Council, 23 (3), 39-41.

Solomon, Paul (1994). Children, Technology, and Instruction: A Case Study of Elementary School Children Using an Online Public Access Catalog (OPAC). SLMR, 23(1). This study explored how children seek out information.

Todd, Ross (April 2003). Irrefutable Evidence. (From EBSCOhost, requires IUPUI login) School Library Journal.

Viadero, D. (4 September 2002). Education department picks groups to develop database of effective practices. Education Week.

Ziegler, M. (2001). Improving practice through action research. Adult Learning, 12 (1), 3-4.

---