Background
Project and Process
Framing the Process
Data Collection and Analysis
Background
In October 1996, the Virginia Adult Educators' Research Network, in
cooperation with the Office of Adult Education and the Performance Indicator
and Data Work Groups, announced the formation of the Assessment Study Group.
Here is how the application described it:
We need teachers willing to share their teaching and assessment strategies,
to explore new activities to improve what they are doing in the classroom,
and to document and report student progress. We are forming a study group,
to work January through June 1997, to:
1. experiment with different types of assessment;
2. discover how to incorporate these assessment techniques into everyday
teaching and lesson planning;
3. describe how each assessment is used in order to build a booklet of
best practices;
4. observe how the use of these assessments affects teaching practices
and program policy;
5. find out if the assessments studied are truly helpful to teachers and
students; and
6. develop ways to document the gathered classroom information and begin
to describe ways to measure the information.
Over 60 practitioners responded to the call for teachers interested in
sharing their current teaching and assessment practices and exploring new
activities about the documentation of adult learning and achievement. The
organizers wanted to assemble a group that represented a variety of instructional
situations and assessment methods. Besides providing some basic information,
applicants had to write about their interests and why they wanted to join.
Here are excerpts from the applications of those selected for the Assessment
Group:
Tom Bello teaches English as a Second Language (ESL) at Langley High School
in Fairfax County. He has worked with adult learners since 1988.
I have done so much work with assessment and would like to continue
to learn more as a member of the study group because I feel that the students'
expressed learning needs play a crucial role in a learner centered curriculum.
I also feel that assessment should be a shared venture between the teacher
and the adult learner. Assessment should measure progress from a positive,
not deficit, model; it should encourage not discourage. Assessment should
be flexible and, as much as possible, appropriate to the individual learner
and to the particular skill being learned. With these ideas, I also bring
to the Assessment Study Group an open mind and a desire to learn more.
Sallie Johnson teaches an Adult Basic Education (ABE) class at the
Adult Learning Center in Lynchburg. She's been working in adult education
for seven years.
Determining the strengths and weaknesses of an individual student is
the foundation of effective teaching strategy. Valid assessments provide
not only a starting point but an on-going indicator of student progress.
Therefore, teaching and assessment are by nature interrelated. I would
welcome the opportunity to experiment with a variety of assessment tools
in addition to those I routinely use. I am interested in determining the
usefulness of specific assessments in facilitating the teaching/learning
process and in studying the validity of such tools for evaluating student
progress.
Linda Mattson is an ABE teacher at the Department of Corrections' Day
Reporting Center in Fairfax County. She has worked with adult learners
for five years.
Participating in the assessment study group would be an outstanding
opportunity for many reasons. Assessment is essential to any adult education
program; through it we are able to custom tailor an educational program
to meet a student's needs. If we can improve the assessment phase of the
Department of Correctional Education's programs, we can improve the entire
program.
Paula Mullins teaches ABE and General Educational Development (GED)
classes at the Wise County Jail and tests clients of Wise County's Department
of Social Services. She has worked with adult learners for nine years.
Assessment and testing are of great interest to me. In the past I have
one field testing for CTB Macmillan/McGraw-Hill in evaluating their new
TABE level E Locator Test, Anchor Test, and Complete Battery (Forms 7 and
8). More recently I did field testing for the Adult Basic Education Technical-Related
Academic Career Competencies (TRACC) program. I worked with Nancy Nolen,
the ABE Coordinator at the Woodrow Wilson Rehabilitation Center during
two different testing years. This was a rewarding experience for me and
my students in Literacy, ABE and Corrections.
Kate Rosenfield teaches adults in a family literacy program in Charlottesville.
She has been an adult education teacher for eight years.
One of the components of [family literacy] programs that always seems
to be lacking is an effective way of evaluating program results. I have
requested information on evaluation, but using assessment tools sometimes
requires training or information that I don't have. I would like to be
a part of the study group in order to research tools for assessment and
to become a better informed and effective teacher. Having credible assessments
will also help in requesting funds to keep our programs active in this
time of disappearing resources.
Amy Stallings teaches at the Lunenburg Correctional Center in Victoria.
She has worked with adult learners for five years.
I taught Learning Disabled students in public schools for 19 years.
I have always used various assessments to diagnose specific problems and
to develop and implement educational plans for each student. Last February,
I began teaching in a correctional center with LIP eligible students [any
inmate whose intake reading scores are below the 8th grade level]. I am
very interested in learning about and developing assessment instruments
that are appropriate for adults. I am presently using parts of various
tests and inventories, in addition to the Test of Adult Basic Education
(TABE), although, I honestly cannot be sure that what I am using is appropriate
for adults.
Gayle von Keyserling teaches a combined ABE/ESL class in Fluvanna County.
She joined the group with one year of experience in adult education.
My interest in assessment tools began with my first ABE class. Actually,
in order to be completely accurate, I must state that my interest was a
result of the frustration I experienced after trying to implement the tools
I had been given for assessment purposes. My first ABE class in our rural
county was comprised of ABE, ESL, and GED level students as well as an
alternative education student from the high school. I knew from the responses
of the students that I needed to do something other than administer the
instruments I had been given. The question was what?
Project and Process
We met as a group three times during the project. Our first meeting,
January 28-30, 1997, aimed to accomplish the following objectives: 1. to
get group members acquainted and familiar with each other's work; 2. to
identify the study's goals; 3. to define the research questions; 4. to
develop a framework for the group's process; and 5. to create a timeline
for the project's activities.
In our opening session, the Study Group members met with Ronna Spacone,
Project Facilitator; Joyce Krech, Acting Coordinator of the Research Network;
adult education leaders and other practitioners. We heard brief presentations
describing interrelated state and national continuous improvement initiatives—all
relative to the study of assessment. Brief descriptions of these initiatives
follow:
Helping Enhance Learning and Instructional eXcellence (HELIX)
HELIX, an upward spiral, symbolizes Virginia's initiative for continuous
improvement of adult education services. HELIX proposes to institutionalize
program improvement by involving all practitioners and learners in the
process of generating, acquiring, using, and disseminating information
about learning and the adult learning system.
The Virginia Adult Education Advisory Committee has five recommendations
for moving adult education in the Commonwealth up the HELIX: 1. Clarify
and document the roles and expectations for all practitioners and programs;
2. Provide full-time instructors and managers at the local and/or regional
levels; 3. Use technology to enhance learning and professional development;
4. Conduct a promotional campaign that accurately presents adult education's
multifaceted delivery system; and 5. Develop a new data system in order
to take advantage of the quality indicator system, i.e. performance indicators,
standards and measures.
Equipped For The Future (EFF)
EFF is the National Institute For Literacy's (NIFL) process for developing
adult education standards and reforming the adult education and lifelong
learning system. Virginia is one of 15 states participating in NIFL's long
range, standards-based initiative. EFF is built on a view of adult education
as preparation for the future, with learning and teaching contextualized
by the key activities that adults typically engage in to satisfy their
real world roles and responsibilities. The process is framed by three critical
adult roles—worker, parent/family member, and citizen/community member—and
four fundamental purposes for learning and literacy. We expect that learning
and literacy will enable us to: 1. access information necessary to being
oriented in the world; 2. have a voice and be heard; 3. make decisions
and be self-sufficient; and 4. build bridges to the future by learning
to learn. EFF's four purposes envelope the multitude of goals that drive
our quest for learning and being successful at school, home, work, and
in the community.
Performance Indicators for Educational Gains
A group of 21 practitioners representing the Virginia Office of Adult
Education, the Performance Indicator Workgroup, and the Data Task Force
met May 23-24, 1996, to explore the issues related to reporting educational
gains and to identify ways to make Virginia's performance indicator and
data management system more meaningful, useful, and credible. The group
validated the significance of the following points: 1. Context matters
in determining educational gains; 2. Educational gains are more than academic
progress; 3. Our accountability system needs to include better ways of
assessing and reporting non-academic gains. The interest in forming a state
assessment study group developed from these proceedings.
Framing the Process
The group process of brainstorming, reworking, and refining was painful at times but eventually productive. The different experiences and personalities of group members made for lively discussion. |
With the above initiatives in mind, our study group reconvened
with Lennox McLendon, Associate Director of Adult and Vocational Education,
in order to get better acquainted, to share additional information about
each other's practices, and to begin defining our work. Our discussion
considered the teachers' individual expectations, their issues and questions
regarding assessment, and how the previous session would influence their
work. It should be noted that consideration of the continuous improvement
initiatives became a factor in the project between the applications process
and our first meeting, thereby broadening the range of possible focal points
and increasing the difficulty with which our group eventually reached consensus
on the study's purpose. Throughout the discussion, we recorded and posted pertinent questions and issues. By the day's end, we had raised 20 questions about assessment, evaluation, reporting, learning, and achievement. (See page 7 for this list.) The objective of the second day's work was to agree on the study's purpose and research questions. The group decided the purpose of the study would be "to capture the reality of current assessment practices". We defined assessment as "those activities used to determine needs, progress, and understanding" and agreed that collecting, analyzing, and using information were integral parts of the assessment process. Three research questions would guide our investigation: 1. How much time does it take to collect, analyze, and use assessment data? 2. What actually happens when we conduct our assessments? 3. How do our assessment practices influence the teaching and learning processes? |
Virginia Assessment Study Group
Questions from the January 28-30, 1997 Meeting
1. What factors contribute to assessment practices being the way they
are?
2. Do our current practices mesh with Equipped For The Future's framework
of the four fundamental purposes for learning?
3. Can we evaluate learning in terms of the extent to which it enables
adults to achieve the four purposes and fulfill real world roles?
4. What is actually happening within our current assessment practices?
5. How is assessment time directed by requirements?
6. How much time does it take to collect, analyze, and share data?
7. What would it take (money, time, and training) to do a really thorough
job of assessment?
8. What knowledge and skills do teachers need to do a high quality job
of assessment and analysis?
9. How does assessment hinder and/or promote the process of teaching and
learning?
10. What kind of learning and achievement are not being measured or reported?
11. How do we know what learners know as a result of our teaching?
12. How can we better match assessment with our learners' goals?
13. How do our teaching styles impact learners?
14. How important are grade levels?
15 How different is what we teach from what we assess?
16 How different is what we assess from what we report?
17. How does the system support or hinder assessment efforts?
18. To what degree do teachers accept that they cannot be all things for
all learners?
19. What role does technology play in assessment practices?
20. To what degree does the classroom setting determine the type of assessments
used?
How much time does it take to collect, analyze, and use assessment data?
|
Our discussion then turned to designing a data collection
system to use throughout the project. We agreed to document routine assessment
practices at least on a weekly basis from February through April. We created
a simple form (see page 10) on which to record data and our observations
and reflections. As we completed the forms, we focused on our three research
question. Research Question 1: How much time does it take to collect, analyze, and use assessment data? On the first section of the data collection form, we recorded the following information: 1. assessment means/instruments; 2. program type and location; 3. purpose of assessment; 4. number of learners involved in assessment; 5. number of learners on site; and 6. amount of time devoted to collecting, analyzing, and using the assessment data. Research Question 2: What actually happens when we conduct our assessments? Here we tried to provide a more descriptive account of what had happened, e.g., where and when the assessment occurred; what was said and done; who was involved; and what the situation demanded of teacher and learner. Background information (relative ircumstances, setting, and conditions) was included to establish context. Research Question 3: How do our assessment practices influence the teaching and learning processes? In the third part of the data collection process, members reflected on the effects of assessment. We wrote about what mattered and what it all meant. We pointed out some of the ways that assessment influences our teaching and asked ourselves what worked, what didn't work, and what could be improved. We reflected on the resources needed to better carry out assessments and what alternative approaches we might use to improve teaching or to enhance learning. Twice during the data collection phase, group members submitted data logs to Ronna, who in turn forwarded copies to all other team members. In April, we stopped collecting data in order to read and begin thinking about what everyone had written. What did it all mean? As we read, we noted text items that answered our three research questions. We also used the following questions to help focus our reading and note making: Are there any problems or issues that need to be addressed? What are the differences and similarities in how group members collect, analyze, and use data? What inferences can we draw from reading and thinking about our various assessment practices? As a result of our reading, we generated a total of 72 questions and issues that we brought to our second meeting on April 18-19. These items helped to focus our discussion and develop the content of our report(s). We decided that each of us would write a separate account of our personal experiences and that Ronna would write about the group process and the group's conclusions. We used our third and final meeting on June 20 to: 1. discuss the report's format; 2. identify final questions and issues; 3. make recommendations for the field's future consideration; and 4. evaluate the process. What follows are the stories of the seven members of the Virginia Assessment Study Group. Our stories represent the diversity of assessment practice in Virginia. We have each included information about the types of classes and learners involved, the physical setting, class schedules, enrollment procedures, and employment situations. We wrote (and re-wrote) these pieces in order to share our understanding of assessment and to identify and explore our own issues and questions. |