Project Summary

Proposal submitted to Network Infrastructure for Education in the area of Evaluation

"Evaluating the Impact of Networking on K-12 Education Reform"

June 14, 1996

1. Statement of Problem and Opportunity

   This proposal describes an evaluation effort to determine whether the introduction of network-based computing is supporting education reform and whether that correlates positively with increases in community-school interaction, and teacher training modelled after Statewide Systemic Initiatives (SSI). We wish to determine those factors that affect organizational change that supports or inhibits reform.

   Numerous educators in school districts throughout the country have embarked on education reform, especially in science and mathematics, with support from the National Science Foundation, state governments and local districts. Simultaneously, and typically as an unrelated effort, many schools have expanded network access and local infrastructure in order to connect to Internet information resources and communication with global, as well as local, community members and groups. The Internet is a natural ally of SSI reform efforts as it facilitates constructivist models, problem-based learning, situated learning modes, and data visualization. It is well argued, however, that the use of technology does not by itself improve learning, but rather must be integrated with reforms in teaching styles and strategies, like those under the SSI reforms, often referred to as learner-centered or constructivist models of education.

   Networking facilitates constructivist strategies with its rich tools and opportunities for student-teacher-community interactions and negotiations to construct meaning. The context of interaction and negotiation is as close as the community and as wide as the world itself. Related, but not necessarily dependent upon constructivist notions, is the belief that students learn best when solving problems in "authentic" or "real life" situations. The power of the Internet to provide a wide range of resources in a rapid manner makes it ideal for problem-based learning activities that require external materials and mentors. The greater accessibility of online simulations of authentic situations, as well as the connection to far-flung and diverse resources (as opposed to, for example, going to the library) expands a learner's and a teacher's actual resource base. The new possibilities that the Internet provides for visualizing scientific data and mathematical models in space offers exciting possibilities for the way in which concepts can be imaged, understood and learned at a level beyond equations (also facilitating a "multiple intelligences" approach to learning).

   Electronic mail and online bulletin boards are the most popular services of the Internet; they are also potentially the most profound, for education, as well as the general community. Computer networks provide a communication medium that has never existed before for small group participation, collaboration and interaction. Recent research shows that in the classroom, online interaction is more than supplemental. Computer mediated communication affects social and psychological aspects of group communication, as well as the quality and extent of discussion, in several specific ways: 1) increases in overall level of discussion of the entire group; 2) increases in student participation, especially from those who do not participate in face to face settings; 3) increases in understanding of material; 4) increases in morale, sense of belonging and motivation; and 5) increases in community interaction and involvement in the educational process, thereby improving the educational experience and facilitating long term sustainability of network resources in the schools.

   There is a body of literature which shows that increased community involvement in education improves education (Latham and Burnahm 1985; Bull et al., 1991; Cohen and Miyake 1986). Set within a community, classroom interactions are potentially extended to any students and teachers in the system, as well as parents, mentors, and others in the community. On the basis of research and experience, we expect community involvement in education to increase and benefit education as a result of networking and school-community programs, such as telementoring (Bull et al 1991; Brienne & Goldman 1990; parent-teacher conferencing, and teacher-teacher conferencing (Dryli 1986; Merseth 1991).

   As schools and communities gain greater access to information networks they will use those networks to link schools and families closer together in support of the teaching and learning process. However, along with the potential for greater linkage comes the potential for increased disparity for children who live in homes without the economic means to provide network access. Where this is true, schools must find creative ways to enfranchise lower socio-economic families and other have-nots in the community.

   In Montgomery County, with support from the US Department of Commerce TIIAP Program, we have established a computer laboratory in the Auburn Middle/High School which not only serves the students and teachers, but also serves the community through evening and weekend Internet training workshops and events. The Auburn community center is a model for our broader vision in which every school in the district (ultimately, the nation) is a resource center for its community.

   The local public library system, the Montgomery Floyd Regional Library (MFRL), serving Montgomery and neighboring Floyd counties, has provided free public access to the Internet (including free electronic mail) since January 1994. MFRL also offers free Internet training workshops to the public about twice a month. These workshops are extremely popular and usually fill up quickly. The infrastructure and training personnel have been supported by the state--Virginia Association of Library Systems--and by the US Department of Commerce TIIAP program. MFRL expects to retain the network librarian and continue the training workshops to serve parents and others in the community who do not yet have or cannot afford a computer in their home. (Please see the MFRL website for more information, including evaluation of training and network usage: http://www.montgomery-floyd.lib.va.us.).

   Technology introduces values and practices that do not always integrate smoothly with those of the school organization. Hodas (1993) attributes the general lack of impact of new technology on teachers, administrators, and students not to implementation failures but rather to a large disparity of the values embedded in the technology from those of the school organizations. The organizational momentum does not support the changes associated with effective deployment of new technologies, with the result that little reform is achieved.

   Resources for education, even moreso than most other sectors, have become increasingly scarce. In order to allocate human and capital resources optimally for education reform, we need rich, generalizable evidence of the circumstances under which reforms achieve maximum effectiveness. Currently, little is known from more than an anecdotal perspective, about the effectiveness of network applications, including community-school interactions, in combination with SSI teaching styles and strategies.

2. Description and Aims of the Project

   The primary aim of this research project is to determine whether the introduction of network-based computing is supporting education reform and whether that correlates positively with SSI training and increased community involvement in education. We wish to determine those factors that positively or negatively affect organizational change that supports reform. For example, among those teachers who are adopting constructivist styles, we would like to know whether these changes are supported organizationally or whether these teachers have simply found the capabilities and resources so compelling that they have pioneered new styles without organizational support.

   To achieve these aims, we will conduct a longitudinal (12 month) evaluation among the teachers, students, administrators, and community members at four school districts in Virginia: Montgomery County, Goochland County, Giles County, and Alexandria City Public Schools. The participating sites are selected on the basis of their commitment to the Statewide Systemic Initiative (SSI) in Virginia, VQuest, and to networking applications in the classroom and with the community. VQuest provides a certain uniformity in approach and goals among Virginia school districts implementing reform. These uniformities facilitate comparison of evaluation results among districts. Taken together, the selected schools provide a rich mix of urban inner city, suburban, and rural environments with an appropriate blend of SSI-trained, and non-SSI-trained teachers, with and without network integration, and varying levels of local community networking.

   All sites will use the same methodolgies, procedures and instruments, selected or adapted collaboratively. Blacksburg, Montgomery County (home of the Blacksburg Electronic Village or BEV) will be the lead site for the project. The Director of Research for the BEV will supervise the project and coordinate activities among sites. Local sites will implement evaluation tasks in their districts, collect data locally, and transfer data (e.g., completed opscan forms and web-based surveys) to the Director of Research in Blacksburg for automated data entry by the Virginia Tech Office of Research and Measurement, as needed. Blacksburg will aggregate data among sites, analyze and write up final results, and disseminate findings on the World Wide Web and in scholarly publications and conferences. Raw data sets and other evaluation materials will be shared openly among participating school districts.

3. III. Prior NSF Support

   The proposed evaluation builds upon two NSF funded projects and a TIIAP demonstration grant, as well as a preliminary evaluation grant from the Council for Library Resources awarded in 1993. The NIE planning grant (entitled "Planning for Virtual Schools in Electronic Villages" no. 9454803) forged a strong collaboration between the Montgomery County Public Schools (MCPS) and the Blacksburg Electronic Village (BEV), which includes Virginia Tech. It served as a catalyst for establishing momentum in the school system toward the infusion of Internet-based computer mediated communication (CMC) into the overall process of schooling. It also served to provide a focal point for planning a broad-scale evaluation of what occurs in the process of schooling when a community becomes an electronic village. Local implementation of the evaluation plan is occuring with support from the Telecommunications and Information Infrastructure Assistance Program (TIIAP) awarded from October 1995 through April 1997.

   Some of the specific outcomes of the planning grant include the pilot implementation of a virtual school, the evaluation plan summarized in this proposal, some preliminary research results discussed in a forthcoming chapter, "Managing the Evolution of a Virtual School," by Roger Ehrich and Andrea Kavanaugh, and "Teaching and Learning in a Virtual School," by Melissa Matusevich in Exit One on the Information Highway: The Design of a Community Network, A. Cohill and A. Kavanaugh (editors), Vinton Cerf, series editor, Artech House (forthcoming 1996). Structured interviews with twelve lead teachers participating in the planning period were conducted in May 1996 and are being analyzed and written up.

   The second NSF funded project is a virtual science lab also funded by NIE and entitled "Human Interface design for Access to Computer and Networked Information" (no. 9554206). Jack Carroll, Professor and Chair, Computer Science Department, at Virginia Tech, is the Principal Investigator; John Burton, inter alia, is a Co-PI. The aim of this project is to design front-end software that allows off the shelf education programs for the Web to become fully interactive in synchronous, real-time mode for science study. It seeks to exploit the potential that the Internet provides for visualizing scientific data and mathematical models in space for the way in which concepts can be imaged, understood and learned at a level beyond equations. The Montgomery County schools are collaborating on the development and testing of this project which got underway in October 1995.

   The preliminary evaluation of the Blacksburg Electronic Village funded by the Council for Library Resources in 1993 engendered a series of foucs groups, survey instruments and an ever-growing database on network users, impacts and demographics in Montgomery County, with assistance from the Center for Survey Research and the Department of Communication Studies, Virginia Tech. Evaluation report and results are online at http://www.bev.net/project/research.

   Two other research projects with whom we have been collaborating closely over the past three years are the Education Infrastructure (Digital Libraries) and Research Infrastructure projects funded by NSF. These projects on education and research infrastructure are directed at technology support and know-how for K-12, in the context of the Blacksburg Electronic Village; Ed Fox and Roger Ehrich, inter alia, Principal Investigators, Computer Science Department, Virginia Tech.

4. IV. Methodology and Workplan

   The evaluation plan is an attempt to provide a design for both a short-term and long-term, multi-dimensional, data collection effort and an organizational framework for interpreting that data. Because the scope of questions can be infinite, we are limiting impact descriptors and guiding questions to four types of learning environments where teachers:

   • integrate learner-centered teaching strategies and network resources into curriculum
   • integrate learner centered teaching strategies, but not network resources, into curriculum
   • do not integrate learner-centered teaching strategies, but do integrate network resources into curriculum
   • do not integrate learner-centered teaching strategies or network resources into curriculum

   The focus of the evaluation is on the process of schooling within the larger context of a networked community, including what was done to promote the infusion of CMC into the selected school districts and what happened as a result of these efforts. The goal of evaluation is to identify change, not the attributional cause of the change. For projects with a scope as broad as the role of networking on education reform, it is not feasible to establish a one-to-one relationship between a specific intervention and the resultant effect. Evaluation becomes a matter of documenting Internet-related practices and shifts in teaching/learning process which emerge simultaneously with the implementation of constructivist teaching strategies, broadly-based Internet capabilities in the schools, and in interaction with the community.

   A triangulation of research methods will be employed to seek greater reliability and validity of results, as well as a richer database for comparison:

   • Participant observation by outside observers
   • Structured interviews and focus groups with administrators, teachers, students, and community members
   • Survey questionnaires for teachers, students, and community members

   We will conduct two surveys of teachers in each of the three participating school districts. We will administer a standard "teaching style" survey which categorizes pedagogical styles according to their similarity to learner-centered teaching style (some teachers may have a constructivist teaching style without having completed SSI training). The teacher pedagogical style instrument will be selected (or adapted, as necessary) from the following candidate instruments: Teacher Attitude Inventory, Mental Measurements Yearbook, Adult Nowicki-Strickland Internal - External Control, Communications Styles Survey, or Mohrman-Cooke Satisfaction Scales.

   The network use and impact survey instrument to be employed will be selected (and adapted, as necessary) collaboratively from the following list of potential instruments:

   • Materials Criteria List, developed by Larry Crum, and used by V-Quest
   • Questionnaires employed by Virginia PEN (developed by Howard "Bud" Cothern);
   • Computer Technology Survey (developed by Mable Kinsey);
   • Teacher use and interest in new media survey (adapted by Larry Arrington);
   • Standardized Test Score Analysis

   We will conduct one mail survey of community members in each school district which will be adapted from the user profile questionnaire and non-user profile questionnaire employed by the Blacksburg Electronic Village for Montgomery County.

   To evaluate the extent to which "technology refusal" at an organizational scale constitutes a hurdle to overall reform efforts, we will conduct structured interviews with elites in the school administration, and review records related to organizational support for integrating network resources and learner-centered teaching strategies.

   We will evaluate the proposition that community involvement improves education by examining traditional vs. new means to involve community. New communication modes (email, newsgroups, interactive video) allow new mentoring relationships to develop among peers, older and younger students, community organizations, and citizens. These relationships enhance learning by fostering cooperation and a spirit of collaboration. Moreover, networking fosters better relationships among teachers, parents and students. To evaluate this community involvement proposition we will be guided by the following questions, methodologies and types of data):

   • In what ways are the schools using technology to reach out to engage parents in the children's learning? (Descriptive study - Mainly Qualitative Data)
   • Are parents and community members coming into the school to obtain access and training on the available technology? (Descriptive study - Mainly Quantitative Data)
   • Are parents more involved in their children's education now than they were before the project began? (Case Studies - Mainly Qualitative Data)
   • Are the disadvantages of students not having access to computer access in their homes being overcome in the participating schools? If so, what strategies are being used to address this issue? (Case Studies - Mainly Qualitative)
   • Have standard measures of student academic achievement improved since networking was initiated? (Descriptive Study - Mainly Quantitative Data)
   • How are non-parent community members being involved in the educational program of the schools? (Case Studies - Mainly Quantitative Data)
   • What barrier to the school becoming a community technology center have been encountered? (Descriptive Study - Mainly Qualitative Data)

   To answer these questions, we will use the following data collection methodologies: focus group interviews, observations, record review, personal interviews, and standardized test score analysis. Professor Stephen Parson, Department of Educational Leadership, College of Human Resources and Education, Virginia Tech will guide the evaluation of the community-school interaction in collaboration with participating school districts and the assistance of a graduate research assistant in the College of Human Resources and Education.

   Education reform initiatives (such as V-Quest in Virginia) have identified several relevant criteria which we will use to guide our assessment of the success of education reform goals, including:

   • widespread adoption of learner-centered instructional strategies (facilitated by increased staff development and better access by teachers to professional development and training, as well as follow-up support through network based communications among administrators, teachers, and parents)
   • increased use of the network to support discovery-based and collaborative learning
   • increased understanding in the community of the curriculum, especially science and math programs (facilitated by increased access to information about the schools), and increased community action in support of education reform (facilitated by increased community access to the network training facilities in the schools and public library)
   • wide use of online materials provided for replication by other districts and communities, and effective integration of materials into local efforts toward education reform and community networking.

   We will pay special attention to the evaluation of differences between rural and suburban responses to education reform and network services. We expect schools and communities--in the heart of Appalachia or the inner city--to be affected differently by networking, and to have very different requirements for acculturation. It is necessary to deploy an interdisciplinary team to evaluate the use and impact of community networking as it occurs in a variety of settings (classroom, public library, government department, residential setting) and in a variety of formats (basic email, newsgroups, automated listservs, database and catalog searches, gopher, WWW).

   An evaluation outline and an evaluation organizer are included in the Appendix. The evaluation outline identifies variables which are incorporated into the evaluation design. The organizer serves two major purposes: 1) it provides a framework within which to identify and categorize evaluation questions, and 2) it serves as a basis for comparing the specific content of surveys and other instrumentation. Its completion is dependent on input from a range of project participants and personnel, who are guided by the literature in this field.

5. Why Us?

   The K-12 school system in Montgomery County, Virginia has been working in close collaboration with the Blacksburg Electronic Village -- a formal partnership among Virginia Tech, Bell Atlantic and the Town of Blacksburg -- over the past three years. With funding from the National Science Foundation (NIE Planning Grant 1994-95), we have planned and pilot tested a virtual school and designed an evaluation plan based on our experiences with teachers and students locally, and on the experiences reported by others in the literature. Additional support from NIE will allow us to broaden the evaluation over time and among other school systems, in order to attain richer, more generaliable results.

   The initial effort to connect the citizens of Blacksburg together and to the world has been a huge success. Over half of the population in the town is directly on the Internet at home, school or at work, and every day, over a thousand people use the BEV home page on the World Wide Web as a jumping off place to obtain services and information. In the rural areas of the rest of Montgomery County, Internet penetration runs about 18%, closer to the pentration rates in the partner districts of Giles County, Alexandria City and Goochland County, and to national averages.

   Rural Montgomery County, Virginia, has very diverse demographics: at one extreme, a nucleus of computer-literate, network-trained individuals typically associated with Virginia Tech and businesses in Blacksburg; at the other, rural residents without telephones who have only a limited sense of connection to an information-based society. The population of the county in 1995 was estimated at 74,000, with an unemployment rate of 4.5%. According to the 1990 census, 19.5% of the county population was below the poverty line. In 1994-95, 33% of county K-12 students received free or reduced price lunches. All of the schools have some access to the Internet, ranging from dial-up connections from the library at 14.4 Kbps to T1 connections (1.44 Mbps) from ethernet-LAN classrooms.

   Alexandria City School System has Northern Virginia's largest minority population which disproportionally reflects the local population. Racially and ethnically, the city is 69.1% White, 21.9% Black, and 9.7% Hispanic, but the school population is 49.1% Black, 18.2% Hispanic, and only 26.5% White. Nearly half of the students in Alexandria public schools come from poorer families. Of the nine school districts in Northern Virginia, Alexandria has the highest number of youth (12-17 years) living below the poverty line. The number of students falling into this category has been growing since 1988. The percentage of students who receive free and reduced price meals has grown from 32% in 1984 to 52% in 1994%. Alexandria City Public Schools operate 12 elementary schools (K-5), 2 middle schools (6-8), one grade nine school, and one large senior high school and an alternative school (9-12). The total student population is 9,656 and is served by 1,004 professional and 579 support staff.

   A new school board, elected in 1994, has made educational technology its top priority; the new superintendent, who began in July 1995, has had extensive experience in educational technology. A very ambitious five-year school technology plan was approved by Alexandria's City Council. Four schools are currently online; all schools will have Internet access in the near future. Alexandria will soon be the nation's first city to be entirely wired with fiber. This infrastructure is expected to be completed and operational in 1997, citywide, under a franchise agreement with the local cable operator, InterJones Cable. The City of Alexandria has enlisted the technical assistance of the Blacksburg Electronic Village in setting up its local community network. Led by Virginia's General Assembly delegate Marion Van Landingham, a delegation representing Alexandria City Council, School Board, and libraries visited Blacksburg on several occasions to obtain training and consultation.

   In Goochland County, every school has a local area network and is connected to the Internet using a frame relay system. The County operates its own servers for the World Wide Web and electronic mail, and maintains a website (http: www.wgland.k12.va.us). Each classroom has three multimedia machines. The Motorola company is testing wireless communications in the county.

   Giles County, adjacent to Montgomery, accesses the Internet through the local phone company, CitizensNet, which offers Internet service regionally. Each school in Giles has been provided a basic account with 30 hours per week. The school district also has the option of using the NASA supported Internet feed from the Governor's School in nearby Pulaski County. Giles has been active in training lead teachers through VQuest in constructivist teaching strategies and styles.

   The proposed evaluation assesses the impact of networking, including increased community-school interaction, on education reform. These developments are sustainable given the growth of Internet use in schools and communities, the diffusion of reform initiatives, and federal and state legislation to improve network infrastructure for education. Our efforts build on the support provided by NSF to the Virginia SSI (VQuest), and Virginia's support for networking in the public schools. VQuest has based its reform efforts on the "Project 2061" recommendations by the American Association for the Advancement of Science (AAAS) and the National Council of Teachers of Mathematics "Standards." These recommendations and standards are consistent with constructivist learning models in the following specific elements: 1) making instruction active and student centered; 2) developing critical thinking and other higher order problem-solving skills; 3) providing the avenue for students to link mathematics, science and technology with authentic or real-life applications; and 4) communicating high expectations for all students. The State of Virginia has also authorized significant expenditures for all public schools to enhance network infrastructure, computer equipment and access to the Internet.

References Cited

Brienne, D. & Goldman, S. (1990) "Network News." Science and Children, 28 (1), 26-29.

Bull, G., Hill, I, Guyre, K., & Sigmon, T. (1991, April). "Building an Electronic Academic Village: Virginia's public education network." Educational Technology, 30-33.

Cohen, M., & Miyake, N. (1986). "A Worldwide Intercultural Network: Exploring electronic messaging for instruction." Instructional Science, 15 (3), 257-273.

Dryli, O. (1986, Winter). "Modems and more: The computer branches out." American Educator: The Professional Journal of the American Federation of Teachers, 10 (4), 14-17.

Ehrich, R. and A. Kavanaugh. "Managing the Evolution of a Virtual School" in A. Cohill and A. Kavanaugh (eds.) Exit One on the Information Highway: The Design of a Community Network. Redding, MA: Artech House (forthcoming 1996).

Ehrmann, S. "Asking the Right Questions: What Does Research Tell Us Abut Technology and Higher Learning?" Change (March/April) 20-27.

Hodas, S. (1993). "Technology Refusal and the Organizational Culture of Schools." Educational Policy Analysis Archives 1, No. 10. Available throught the listserver edployar@asuvm.inre.asu.edu.

Johnson, J.R. (1989). Technology: Report of the Project 2061 Phase I Technology Panel. Washington D.C.: American Association for the Advancement of Science.

Kozma, R. and J. Johnston. 1991. "The Technological Revolution Comes to the Classroom," Change, January/February 1991: 10-23.

Latham, G. & Burnham, J. (1985). "Innovative methods for serving rural handicapped children." School Psychology, 14 (10), 438-443.

Merseth, K. (1991). "Supporting Beginning Teachers with Computer Networks." Journal of Teacher Education, 42(2), 140-147.

National Commission on Excellence in Education (1983). A Nation at Risk.

Neuman, R. 1993. The Future of the Mass Audience. New York: Cambridge University Press.

Norman, D. A. and Spohrer, J. C. (1996). "Learner Centered Education." IEEE Computer, 39 (4), 24-27.

Ruberg, L. 1994. Computer Mediated Communication Environment. Virginia Polytechnic Institute and State University, unpublished Ph.D. dissertation.

Soloway, E. 1995. "Beware, Techies Bearing Gifts," Communications of the ACM. Vol. 38, No. 1: 17-24

Sproull, L. and S. Kiesler. 1991. Connections: New Ways of Working in the Networked Organization. Cambridge, MA: MIT Press.