BRT

funding sources

Since 1985, Behavioral Research and Teaching has been the recipient of over 26 million dollars in federal and state level research, training, and development funds. An overview of current projects is provided below.

This proposal focuses on enhancing the statewide longitudinal data system that has been developed over the last decade in Oregon. Currently, the state has been successful with legislative funding and federal awards in developing a centralized and standardized system for organizing and storing student records and transcripts in statewide data warehouses. At the individual professional use level, educators now have a unique staff identifier that allows them access to a secure system for focusing on school improvement; at the institutional support level, common file formats now allow integration of records from K-12, community colleges, and the Oregon University System (OUS). While the data structures and integration protocols appear to be in place, to be maximally effective in using student performance and progress data to drive educational decisions, three enhancements are needed. First and foremost, classroom data need to be added as a critical ingredient to complement the data elements currently present. While demographic data are important as blocking factors for analyzing and reporting data, they are not changeable and therefore limited as targets for instructional design. Although statewide test data are present, performance is distal to daily classroom events and progress is eventual in showing change. Second, better empirical analyses need to be conducted to create first order change that can lead to more localized, second order change. Recent advancements in statistical analyses that take into account the nested nature of performance using hierarchical and growth models can now be used to better understand the complex nuances of student performance and progress. Third, teachers need to be better prepared to not only mine the data but also use it in finding systematic patterns that have curriculum and instructional implications. This teacher training is best done at the beginning of their careers so that teaching and learning can become integrated in a holistic approach. Given the explosion in creation, transfer, storage, and reporting of information with advancements in computing technologies, teachers need to learn how to place learning first so that teaching can be appropriately developed and delivered. Over the next three years, we propose to enhance the data integration system in the statewide data warehouses by incorporating classroom performance and progress measures, build large-scale data sets for policy analysis and development, and create pre-service teaching materials to use the classroom performance and progress measures. We can use an extant curriculum-based measurement (CBM) system that has been developed at the University of Oregon (easyCBM.com). This system is operational and currently used by teachers in all 50 states across the country with districts having the option of both benchmarking testing (as predictors of performance on the statewide test) and progress monitoring (as students are identified at risk of failure and require more formative evaluation). In this project, we will adapt the easyCBM data warehouse system to allow districts to control exchange of student demographic information and state scores into their local district easyCBM site. Once these data have been integrated into local (district) data warehouses, the Oregon Department of Education (ODE) can integrate them across regions and begin the process of file creation for large-scale data mining by policy analysts. Using secure access control points and systematic file structures, university researchers can work with ODE to conduct specific studies of longitudinal data sets to better understand various district and state policy initiatives. Finally, using previous work conducted at the University of Oregon, a series of web-based teacher training modules will be developed for incorporation into pre-service programs within the six Oregon University System (OUS).

The purpose is to develop a comprehensive screening and progress monitoring system in reading appropriate for use with students in grades 3 through 5. The online assessment system will include a variety of reading measures, as well as an online training component on test administration and scoring. We propose adding these measures to an online elementary school reading assessment system developed with prior federal grant funding. This already-existing technological infrastructure will enable us to surmount one of the most significant barriers to the use of progress monitoring assessments: the time and logistics previously associated with test administration, scoring, and record keeping. Specifically, in our first year we propose to develop screening and progress measures in silent reading fluency, vocabulary, and comprehension appropriate for this age group, using Item Response Theory (IRT) scaling to develop comparable alternate forms. The use of IRT during test construction sets these measures apart from more traditionally designed formative assessments. They stand out for their sensitivity in monitoring growth, the stability of alternate forms, and for the information they provide about the full range of student reading skill: from the skill of developing automaticity in reading fluency to the end goal: understanding word meaning and comprehension of narrative and expository text. In year 2, we implement the measures for decision-making within a response-to-intervention model. We plan to have teachers access various instructional interventions as they first identify students at risk of failure and then monitor their progress over time. We plan to develop the infrastructure for implementing both the data collection and display using computer-based tests. With silent reading fluency, we plan to have measures that are closely related to oral reading fluency as well as vocabulary and comprehension. In vocabulary, we plan to have two forms that measure recognition of synonyms and use of words in context (with a maze strategy that requires students to complete sentences by filling in the correct option). In comprehension, we plan to use short narrative and expository text structures to assess both literal and inferential comprehension. In all three (skill) areas, we plan to display student performance compared to each other (nor-referenced), as reflections of specific skills (criterion-referenced), and as progress over time (individually referenced). The settings are classrooms in districts near the university. We estimate approximately 25,000 students to be influenced by this research. We have three outcomes from this project: (a) an IRT-scaled benchmarking and progress monitoring system that extends beyond oral reading fluency and is thus capable of being delivered as part of a computer-based test administration, (b) a data collection system for teachers to use in developing effective instructional programs, and (c) a web-based system for displaying instructional interventions so teachers can share what works.

The purpose of this grant is to develop and validate an assessment extended seamlessly from general education large-scale tests that also reflects growth within and across elementary and middle school on grade level content standards. We accomplish this outcome by (a) developing reliable alternate forms of tasks that are aligned with grade level content standards, (b) vertically scaling them across grade groups in two bands (elementary and middle), (c) collecting criterion-related evidence to anchor interpretation to classroom practices and (d) using growth modeling to document change, both across and within grades. Using a validation argument, we make the claim that with intensive instruction, this group of students can indeed change their learning trajectories to master grade level content. Our focus is on collecting content-related evidence, reliability, criterion-related evidence, and change over time to support this claim. We use four primary data analytic strategies: (a) universal design to create measures accessible for the widest range of students followed by a formal alignment analysis, (b) item response theory (IRT) to develop assessments and to create vertically equated scales, (c) IRT and classical test theory methods to evaluate the reliability and validity of the measures, and (d) growth curve modeling to understand the learning trajectories, with the criterion-related evidence helping us understand why students may or may not improve.

This Progress Monitoring Center is designed to institute the next generation of curriculum-based measurement for school districts. By combining the test development technology of Item Response Theory (IRT) with the measurement sampling plans of curriculum-based measurement, this project can provide school districts formative assessments in reading with strong technical adequacy. First, we plan to develop five skill area inventories in reading and scale them using Item Response Theory (IRT) for developing alternate forms of equivalent difficulty: (a) letter names and sounds, (b) phoneme segmentation and blending, (b) word and sentence reading, (d) oral reading fluency of passages, and (d) comprehension. As part of the measurement system, we can then develop a screening instrument to develop norms for the district in grades K-4 in the fall and in the spring Second, we propose building a data entry node on this web site where teachers can enter the values from their frequent measures and receive basic output (summarizing performance over time and reporting on various characteristics of performance like slope, variability, level change, and overlap). We plan to institute frequent measures in each building with 2-3 teachers, targeting students who are below the 10th percentile rank. Third, we plan to develop training modules on the administration and scoring of the CBMs in these five skill areas and provide both teacher and students materials on a web site so they can easily download all measures. In this training, teachers are taught basic administration and scoring rules with short video clips and then tested on their proficiency by having them score several samples; their score is tracked and they are provided feedback on their performance.

This project focuses on Instruction, Assessment, and Standards as well as Information Management to Support Decision Making. The application is designed to assist state departments in organizing measurement of state standards into a cohesive framework for decision-making that is responsive to the diversity of students with disabilities. In this project, we accomplish three outcomes so that the process of large-scale testing can become more meaningful for students with disabilities. First, we develop a computerized a model of alignment that includes several dimensions allowing teachers to understand student's specific skills relative to state standards. We focus on the concurrence between the measures and the standards, the depth of knowledge being assessed in the measures, the range of knowledge being assessed (relative to the standards), and the balance of representation within with measures that are represented in the standards. Second, we assist teachers in deciding how students should participate in taking the state test in the standard manner, with accommodations, with modifications, or by taking an alternate assessment? Rather than basing this decision on a subjective judgment of prior classroom applications (which may be unreliable), we provide a computerized curriculum-based measurement (CBM) system to help teachers decide the optimal manner for participation. Third, we provide outcome reports that show relationships between various measures of achievement that are aggregated in a meaningful manner. We computerize a data entry and reporting system that allows outcomes to reflect both an idiographic and nomothetic perspective.

This project provides teachers a system for making decisions on how students should participate in large-scale testing programs (standard test, 2%, 1%) and what types of accommodations they should receive. Assessment: The process for determining how students should participate in large-scale assessments is problematic. With five possible options, what data sources and criteria should IEP teams use. Almost no research exists on making decisions for the 1% other than that the student needs to be with the most significant cognitive disabilities; however, most states disavow (and appropriately so) making decisions on the basis of a disability. In the white paper, we articulated a series of questions that IEP teams had to address about receiving instruction in grade level content, access skills to this content, use of communication systems, and degree of scaffolds needed for participation. Accommodation: A need exists for accommodations in large-scale testing, though it is somewhat uncertain how and why they appear to be effective and with which students. The empirical support for them is inconsistent within and across subject areas and even though they sometimes are effective, they also appear to be either inert (not work for anyone) or overly effective (work for everyone). It is not yet possible, therefore, to simply move research to practice in adopting wide scale adoption of specific accommodations that have passed the test of replicable empirical support. Yet, large-scale testing requires their application. To bridge this gap, the research on accommodations has begun to focus on how teachers make the decision to recommend specific changes in testing.

Alternate achievement standards are designed to enable inferences to grade level expectations that have been extensively prioritized but maintain high expectations for progress in the general curriculum and assume student performance is contingent on having the supports specified for the assessment. Inferences are stipulated. These assessments reflect changes that can be made in the levels of support or in the breadth, depth, or complexity of the standards that are being assessed. Support refers to the types of scaffolds, prompts, and assistive technologies used in the administration of the assessment. Breadth refers to the extensiveness of content and skills linked to the standards, curriculum, or assessments; depth refers to the level of complexity of cognitive functioning i.e., factual-associative, conceptual-hierarchical, principles-causative to be successful on performance standards. In Oregon, each test includes both teacher Administration and Scoring Protocols and Student Materials. Each test is comprised of 11 tasks. Task 1 contains 10 prerequisite items and Task 2 through Task 11 each contains five content items. In Alaska, each test includes skills in subject areas that are linked to Extended Grade Level Expectations (ExGLEs) that articulate pre-requisite skills for success on grade level standards; Expanded Levels of Support represent even more basic skills for functioning with others. In both states, the emphasis is on applying current measurement and analysis models to understanding accountability systems for a population that has not ever been included. We address issues of validity as outlined in the 1999 Standards for Psychological and Educational Research: content-related evidence that addresses alignment with standards, the internal structures of the instruments, criterion-related evidence that provides relations with external measures, and finally, response processes that consider the manner of interacting with the tests.