«What the NAEP Civics Assessment Measures and How Students Perform by Peter Levine with support from the S.D. Bechtel, Jr. Foundation January 2013 ...»
What the NAEP Civics Assessment Measures and
How Students Perform
by Peter Levine
with support from the S.D. Bechtel, Jr. Foundation
Only eight states currently test their students on American government or civics
(usually as part of a much broader social studies test), and so relatively little is
known about young people’s civic knowledge, skills, behaviors, and values.1 Given
the paucity of state data, the federal National Assessment of Educational Progress
(NAEP) in Civics receives predominant attention. The fact that only about one quarter of students typically reach the “proficient” level on the NAEP Civics assessment is probably cited more than any other statistic about civic education, and it is often used to support proposals for adding civics requirements.
Indeed, civic education deserves increased attention, and students’ knowledge may be problematic, but these interpretations of the NAEP are based on misunderstandings. This fact sheet explains how to interpret its results.
Background on the NAEP Civics assessment The NAEP system was designed to give a regular national “report card” on students’ academic proficiency to inform citizens and policymakers. Each NAEP is a test-like assessment on one subject given to randomly sampled American students. In Civics, the NAEP assessment is now administered to more than 21,000 students every three years. Equal thirds of the participants are fourth graders, eighth graders, and 12th-graders.
Compared to other prominent tests and surveys in civics, the NAEP is the most carefully designed and validated and has the largest national sample. Whereas existing state civics tests rely exclusively on multiple-choice questions, the NAEP Civics assessment also includes short essays, which are better measures of advanced skills.
Although participation is voluntary, response rates are generally high. Of the fourth graders in public schools who were asked to participate in 2010, 99% did so, along with 92% of 12th graders in public schools (but just 62% of 12th graders in private schools).2 A more significant source of bias may arise from a lack of motivation. The assessment has no stakes for individuals, schools, or districts; in fact, teachers and students are not even told their own performance. Given the lack of consequences, it is very unlikely that students study or prepare for the NAEP. Teachers have no reason to align their curricula to the NAEP’s content, unless it happens to match requirements of their state or district.
The NAEP scores do not indicate whether students’ civic knowledge is sufficient Typically, the release of NAEP Civics results is treated as evidence that students know far too little about civics. The most recent results (from the 2010 assessment) generated an article in The New York Times entitled “Failing Grades on Civics Exam Called a ‘Crisis.’”3 Only 24% of 12th graders were deemed proficient, and a similar number reached proficiency in 4th and 8th grade.4 Many reporters cited these statistics, along with specific questions on which students performed poorly. The Times story began, “Fewer than half of American eighth graders knew the purpose of the Bill of Rights.” These interpretations of the NAEP are misleading. The test is designed to produce the overall results that it yields. When the current NAEP Civics Framework was developed in 1998, a committee of teachers and other knowledgeable citizens decided how difficult each proposed question ought to be for students at each grade level, and then decided what overall score should qualify a student as having “basic,” “proficient,” or “advanced” knowledge. If members of the committee had agreed on a more—or less—demanding idea of what qualifies as success in civics, they would have set the cutoff scores either higher or lower, and those decisions would have been equally valid. The committee was informed by empirical data, but ultimately it had to make a value-judgment about what questions to ask how much knowledge is satisfactory.5 Each subsequent NAEP Civics assessment has then been carefully constructed to be as equally difficult as the 1998 assessment. Thus 24% percent of 12th graders were rated “proficient” in 2010 because the NAEP 12th grade test had been designed to yield roughly that proficiency rate.
On the most recent NAEP Economics assessment (2006), 42% of seniors reached the proficient level—18 points higher than the scores for the most recent NAEP Civics.6 That does not imply that students are more knowledgeable about economics (which receives less attention than civics in most American schools). It simply means that the economics committee defined “proficient” less rigorously than the civics committee did. Neither committee was right or wrong.
For a citizen or policymaker who wants to decide whether students know enough, there is no substitute for looking at individual NAEP items and making an independent decision about whether most students at a given grade level should know the answers. Citizens may find some NAEP items surprisingly difficult, but they may think that other items are easy and should yield mostly correct answers. At the end of this fact sheet, some examples of items are provided as illustrations.
What the NAEP results do show Although overall scores on the NAEP Civics assessment do not tell us objectively how well students perform, the assessment is highly informative if interpreted
correctly. The results can be used to learn:
1. Which students perform better and worse than the norm for their grade. The National Center for Education Statistics (NCES) reports that White students, female students, students whose parents had completed college, and students not eligible for free or reduced-price lunch scored better than their counterparts on at least one of the grade levels of the NAEP Civics in 2010.
For more detail on demographic differences in NAEP outcomes, see the NCES’s The Nation’s Report Card: Civics 2010. Other research finds that students from more advantaged background receive the most intensive and engaging opportunities to learn civics in schools; unequal inputs may in part explain unequal results.7
2. How students’ knowledge has changed over time. Some questions have been repeated on each NAEP Civics assessment since 1998 to establish trend lines. The difficulty of the whole test is then calibrated by comparing performance on the other items to these repeated questions. Although the way the NAEP is designed makes it resistant to change in the mean scores, some shifts have occurred. At 4th grade, the trend has been upward since 1998; at 12th grade, scores have declined since 2006. Those findings are surprising in light of the fact that time devoted to social studies has shrunk in the early grades—under pressure from testing and standards—but it has increased in high school.8 For details on trends in scores, see The Nation’s Report Card: Civics 2010.
3. Which educational practices are related to higher scores. Students who take the NAEP are asked about the frequency of various instructional practices, from completing worksheets in class to conducting debates. Their teachers are also surveyed about these practices, giving two separate perspectives on what happens in the classroom. In a subsequent fact sheet, CIRCLE will explore how instructional practices relate to students’ scores on the NAEP.
4. How well students understand various specific topics. By examining performance on individual items, we can learn more about which topics American students are mastering and which ones may need more attention.
We will turn to that issue at the end of this fact sheet. But first, it is important to understand what the NAEP measures.
What the NAEP Measures
According to the Framework document that has governed the NAEP since 1998,9 20% of the NAEP Civics assessment should focus on “American constitutional government and its history” and the “distinctive characteristics of American society and American political culture” that “are linked to American constitutional democracy.” Another 25% of the items should assess students’ understanding of how “the government established by the Constitution [embodies] the purposes, values, and principles of American democracy.” Thus a total of 45% of the test should relate to the US constitutional system, defined broadly.
Twenty-five percent of the items should involve the role of citizens. Because the NAEP may not assess students’ actual civic participation, students are instead asked to identify and explain actions taken by citizens in historical or hypothetical situations. For example, students cannot be asked whether they volunteer, but they could be asked why some citizens volunteer. Another 10% of the items must address the nature of civic life, politics, and government; the remaining 20% of the test investigates the role of the US in the world.
Perhaps the most succinct summary of what the NAEP measures comes from an
official definition of civics “proficiency” at the 12th-grade level:
Twelfth-grade students performing at the Proficient level should have a good understanding of how constitutions can limit the power of government and support the rule of law. They should be able to describe similarities and differences among constitutional systems of government, and they should be able to explain fundamental American democratic values, their applications, and their contribution to expanding political participation. They should understand the structure of American government and be able to evaluate activities of political parties, interest groups, and media in public affairs. They should be able to explain the importance of political participation, public service, and political leadership. They should be able to describe major elements of American foreign policy and the performance of major international organizations.10 What qualifies as a “good understanding” is a matter of judgment, not statistical fact.
Fig. 1: Comparing the Civic Mission of Schools, state standards, and the NAEP Figure 1 puts those specifications in comparative context. The NAEP’s breakdown is shown in the right-hand column. At the left are the outcomes deemed essential in the Civic Mission of Schools report, which was written by 60 diverse experts and led to the formation of the Campaign for the Civic Mission of Schools.11 The outcomes that
they selected were:
civic skills (such as working with other people to address a public problem), civic values (such as concern for the common good and the rights of others), civic engagement (such as voting and volunteering), and civic knowledge, including knowledge of law, history, politics, and community life.
The Civic Mission of Schools did not suggest how these four categories should be weighted, so they are shown as equally important in the left-hand column.
Immediately, one can see that the Civic Mission of Schools outcomes differ from the measured topics in the NAEP Civics Assessment. The NAEP is weighted far more heavily toward knowledge, and especially knowledge of the US Constitution. In fact, the entire NAEP could be described as a test of knowledge, although students are asked some factual questions about civic engagement under the heading “role of citizens” and may be asked to display such academic skills as interpreting a cartoon or giving reasons for political position. Since the NAEP is a paper-and-pencil test taken privately by individual students, it does not directly assess students’ ability to collaborate or deliberate in groups. By statute, it may not ask about behaviors outside of school or about values.
The middle column indicates the topics that are typically emphasized in state standards. All states have standards for civics, and most draw on the voluntary National Curriculum Standards for Social Studies from the National Council for the Social Studies.12 CIRCLE recently reviewed all the states’ standards and identified specific elements that matched the categories in the National Curriculum document.
In figure 1, those categories are shown in proportion to how many states require them. For example, all states and the District of Columbia have standards for “civic skills,” but only 12 states have standards about “individual identity and development,” so civic skills are shown as a larger portion of the column. Overall, the central column of the graph gives a rough, aggregate portrait of what American students are expected to learn.
Comparing the three columns reveals that:
“Civic skills” are emphasized in the Civic Mission of Schools report and are mentioned in most state standards, but they do not represent one of the tested categories in the NAEP. (Some NAEP items arguably require civic skills, such as interpreting a speech, to answer correctly.) “Civic engagement” is cited as an explicit outcome in the Civic Mission of Schools and is often included in state standards, but it is not directly measured in the NAEP.
One fifth of the NAEP is devoted to international or global issues, which were not mentioned at all in the Civic Mission of Schools and are not common in state standards.
The “role of citizens” represents 25% of the NAEP and overlaps with two categories in state standards (“civic skills” and “civic ideals and practices”) as well as two outcomes in the Civic Mission of Schools framework (“civic skills” and “civic values/dispositions”). However, the NAEP items on this topic are much more academic, consisting of questions about how and why citizens act, instead of measuring actual actions by students.
The US Constitution and the foundations of the US political system are much more prominent in the NAEP than in state standards or the Civic Mission of Schools.
Is the NAEP aligned with what students learn?
The NAEP is a federal assessment with no consequences for students or schools. In contrast, state standards are legal documents adopted by state governments that are meant to govern curriculum, instruction, textbooks, and tests. Nevertheless, the NAEP Framework may reflect the typical emphasis in classrooms better than the state standards do.
In a large, nationally representative survey of 4,483 young adults conducted in 2012, CIRCLE found that 44.9% of young adults remembered “the Constitution or the U.S.