Translate

Monday, February 27, 2012

Special Ed. Vouchers May Open Doors for Choice

Special Ed. Vouchers May Open Doors for Choice

Meet voucher supporters' new fellow strategists: students with disabilities.

Creating private school vouchers for special education students—programs that are largely unchallenged in court, unlike other publicly financed tuition vouchers—can be the perfect way to clear a path for other students to get school options, according to school choice proponents.

With this approach, "there is more success legislatively," said Malcolm Glenn, a spokesman for the Washington-based American Federation for Children. The group advocates school choice, focusing its efforts on tuition vouchers and scholarship tax-credit programs.

'Value Added' Concept Proves Beneficial to Teacher Colleges

'Value Added' Concept Proves Beneficial to Teacher Colleges

Only because there is a poverty in imagination and a real lack of understanding of the limitations of this approach. Oh yes, and a desire to control teachers, corporatize, privatize, etc.

-Angela

Friday, February 17, 2012

Angela Valenzuela: A superintendent for DISD’s English-language learners

Angela Valenzuela: A superintendent for DISD’s English-language learners

Angela Valenzuela: A superintendent for DISD’s English-language learners | Dallas Morning News Opinion and Editorial Columns - Opinion and Commentary for Dallas, Texas - The Dallas Morning News

Published: 16 February 2012 06:01 PM

In light of the very high representation of English language learners in the Dallas Independent School District — an astounding 37.8 percent, more than twice the state average — the next superintendent should definitely be someone who knows and understands the research pertaining to this group of students.

This research consists of bilingual education, language acquisition, quality English as a Second Language programs and instruction, student placement and program evaluation. The new superintendent should bring experience in improving and enhancing ESL programs and services. Will transitional bilingual education models be maintained? If so, will these include both early- and late-exit models? Will the new superintendent promote dual-language instruction? If so, what models would apply best districtwide, given a context of school segregation by race or ethnicity and class?

Regarding bilingual education, a study commissioned by the Texas Education Agency in 2000, the Texas Successful Schools Study: Quality Education for Limited English Proficient Students, showed that English-language learners who remained in the bilingual program until they were designated as “English proficient” met or exceeded the performance of students in the all-English program in the same grade levels and at the same schools. Such findings are typical in research on effective bilingual education programs. This debate is more about politics than evidence.

For nonbilingual education and non-ESL personnel, the superintendent needs to consider how general education teachers can be better prepared to serve English language learners in their classrooms. Improving collaboration between ESL and general classroom teachers is often overlooked, yet this is so vital to a welcoming, positive school environment.

The benefits of bilingualism and biliteracy are widely recognized for children’s cognitive development. Both promote academic achievement, cognitive flexibility and problem-solving capacities that are not enjoyed by monolingual speakers. Indeed, all children stand to benefit from knowing a second language at an advanced level. To this end, what we need are rich opportunities for youths both at the elementary and secondary levels that promote the development of the students’ native language to keep pace with their academic development in the English language.

At the district level, central office and campus administrators also need to be prepared to serve the needs of culturally and linguistically diverse students. Clearly, qualified teachers and administrators of English language learners are needed in great numbers, so what plans will the superintendent have to prepare, recruit and retain them?

Standardized tests provide metrics for determining program success. However, the information they provide is not only limited but is also language-dependent, in the case of English-language learners. Hence, what other criteria might this new superintendent use to gauge progress toward closing the achievement gap?

The new superintendent needs to promote high expectations and academically challenging instruction to prepare these youths for college. Minimum-track classes that lead to a minimum diploma should be avoided for all students in general and for English language learners in particular. Unfortunately, being an English learner and a student with college aspirations tend not to go hand in hand. How, then, can the next superintendent work to reverse such mind-sets?

Finally, it is not enough for our children to be bilingual. For them to function at advanced, professional levels in our global society, they must also be biliterate. They must command the ability to read and write at sophisticated levels across all content areas in powerful writing contexts for authentic audiences and purposes.

The future superintendent should promote bilingual or ESL programs that are well-funded, staffed and designed if they are to genuinely address the very achievement-gap problem that otherwise bedevils most superintendents and district leaders.

Angela Valenzuela is a professor in the College of Education at the University of Texas at Austin. Her email address is valenz@austin.utexas.edu.
Our goal: To advocate hiring a strong change agent at DISD and for instituting reforms with records of success.

Our work so far: Since September, we’ve been publishing exclusive commentaries by national education experts on DISD’s future and editorials offering our recommendations.

Coming Sunday: Our editorial on a superintendent wish list

READ the Tactics for a Turnaround series so far. dallasnews.com/turnaround

Thursday, February 16, 2012

Leading mathematician debunks ‘value-added’

Excellent analysis of "value-added" modeling and why it's a bad idea.  Quotes from within:  

"When value-added models were first conceived, even their most ardent supporters cautioned about their use [Sanders 1995, abstract]. They were a new tool that allowed us to make sense of mountains of data, using mathematics in the same way it was used to understand the growth of crops or the effects of a drug. But that tool was based on a statistical model, and inferences about individual teachers might not be valid, either because of faulty assumptions or because of normal (and expected) variation."

"People recognize that tests are an imperfect measure of educational success, but when sophisticated mathematics is applied, they believe the imperfections go away by some mathematical magic. But this is not magic. What really happens is that the mathematics is used to disguise the problems and intimidate people into ignoring them—a modern, mathematical version of the Emperor’s New Clothes."

"Unfortunately, VAM proponents and politicians have framed the discussion as a battle between teacher unions and the public."

"Of course we should hold teachers accountable, but this does not mean we have to pretend that mathematical models can do something they cannot. Of course we should rid our schools of incompetent teachers, but value-added models are an exceedingly blunt tool for this purpose. In any case, we ought to expect more from our teachers than what value-added attempts to measure."

-Angela


05/09/2011

Leading mathematician debunks ‘value-added’ http://www.washingtonpost.com/blogs/answer-sheet/post/leading-mathematician-debunks-value-added/2011/05/08/AFb999UG_blog.html

--
This was written by John Ewing, president of Math for America, a nonprofit organization dedicated to improving mathematics education in U.S. public high schools by recruiting, training and retaining great teachers. This article originally appeared in the May Notices of the American Mathematics Society. It gives a comprehensive look at the history, current use and problems with the value-added model of assessing teachers. It is long but well worth your time.

By John Ewing

Mathematicians occasionally worry about the misuse of their subject. G. H. Hardy famously wrote about mathematics used for war in his autobiography, A Mathematician’s Apology (and solidified his reputation as a foe of applied mathematics in doing so). More recently, groups of mathematicians tried to organize a boycott of the Star Wars [missile defense] project on the grounds that it was an abuse of mathematics. And even more recently some fretted about the role of mathematics in the financial meltdown.

But the most common misuse of mathematics is simpler, more pervasive, and (alas) more insidious: mathematics employed as a rhetorical weapon—an intellectual credential to convince the public that an idea or a process is “objective” and hence better than other competing ideas or processes. This is mathematical intimidation. It is especially persuasive because so many people are awed by mathematics and yet do not understand it—a dangerous combination.

The latest instance of the phenomenon is valued-added modeling (VAM), used to interpret test data. Value-added modeling pops up everywhere today, from newspapers to television to political campaigns. VAM is heavily promoted with unbridled and uncritical enthusiasm by the press, by politicians, and even by (some) educational experts, and it is touted as the modern, “scientific” way to measure educational success in everything from charter schools to individual teachers.

Yet most of those promoting value-added modeling are ill-equipped to judge either its effectiveness or its limitations. Some of those who are equipped make extravagant claims without much detail, reassuring us that someone has checked into our concerns and we shouldn’t worry. Value-added modeling is promoted because it has the right pedigree — because it is based on “sophisticated mathematics.”As a consequence, mathematics that ought to be used to illuminate ends up being used to intimidate. When that happens, mathematicians have a responsibility to speak out.

Background
 
Value-added models are all about tests—standardized tests that have become ubiquitous in K–12 education in the past few decades. These tests have been around for many years, but their scale, scope, and potential utility have changed dramatically.

Fifty years ago, at a few key points in their education, schoolchildren would bring home a piece of paper that showed academic achievement, usually with a percentile score showing where they landed among a large group. Parents could take pride in their child’s progress (or fret over its lack); teachers could sort students into those who excelled and those who needed remediation; students could make plans for higher education.

Today, tests have more consequences. “No Child Left Behind” mandated that tests in reading and mathematics be administered in grades 3–8. Often more tests are given in high school, including high-stakes tests for graduation.

With all that accumulating data, it was inevitable that people would want to use tests to evaluate everything educational—not merely teachers, schools, and entire states but also new curricula, teacher training programs, or teacher selection criteria. Are the new standards better than the old? Are experienced teachers better than novice? Do teachers need to know the content they teach?

Using data from tests to answer such questions is part of the current “student achievement” ethos—the belief that the goal of education is to produce high test scores. But it is also part of a broader trend in modern society to place a higher value on numerical (objective) measurements than verbal (subjective) evidence. But using tests to evaluate teachers, schools, or programs has many problems. (For a readable and comprehensive account, see [Koretz 2008].) Here are four of the most important problems, taken from a much longer list.

1. Influences. Test scores are affected by many factors, including the incoming levels of achievement, the influence of previous teachers, the attitudes of peers, and parental support. One cannot immediately separate the influence of a particular teacher or program among all those variables.

2. Polls. Like polls, tests are only samples. They cover only a small selection of material from a larger domain. A student’s score is meant to represent how much has been learned on all material, but tests (like polls) can be misleading.

3. Intangibles. Tests (especially multiple-choice tests) measure the learning of facts and procedures rather than the many other goals of teaching. Attitude, engagement, and the ability to learn further on one’s own are difficult to measure with tests. In some cases, these “intangible” goals may be more important than those measured by tests. (The father of modern standardized testing, E. F. Lindquist, wrote eloquently about this [Lindquist 1951]; a synopsis of his comments can be found in [Koretz 2008, 37].)

4. Inflation. Test scores can be increased without increasing student learning. This assertion has been convincingly demonstrated, but it is widely ignored by many in the education establishment [Koretz 2008, chap. 10]. In fact, the assertion should not be surprising. Every teacher knows that providing strategies for test-taking can improve student performance and that narrowing the curriculum to conform precisely to the test (“teaching to the test”) can have an even greater effect. The evidence shows that these effects can be substantial: One can dramatically increase test scores while at the same time actually decreasing student learning. “Test scores” are not the same as “student achievement.”

This last problem plays a larger role as the stakes increase. This is often referred to as Campbell’s Law: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to measure” [Campbell 1976]. In its simplest form, this can mean that high-stakes tests are likely to induce some people (students, teachers, or administrators) to cheat ... and they do [Gabriel 2010].

But the more common consequence of Campbell’s Law is a distortion of the education experience, ignoring things that are not tested (for example, student engagement and attitude) and concentrating on precisely those things that are.

Value-Added Models

In the past two decades, a group of statisticians has focused on addressing the first of these four problems. This was natural. Mathematicians routinely create models for complicated systems that are similar to a large collection of students and teachers with many factors affecting individual outcomes over time.

Here’s a typical, although simplified, example, called the “split-plot design.” You want to test fertilizer on a number of different varieties of some crop. You have many plots, each divided into subplots. After assigning particular varieties to each subplot and randomly assigning levels of fertilizer to each whole plot, you can then sit back and watch how the plants grow as you apply the fertilizer. The task is to determine the effect of the fertilizer on growth, distinguishing it from the effects from the different varieties. Statisticians have developed standard mathematical tools (mixed models) to do this.

Does this situation sound familiar? Varieties, plots, fertilizer ...students, classrooms, teachers?
Dozens of similar situations arise in many areas, from agriculture to MRI analysis, always with the same basic ingredients—a mixture of fixed and random effects—and it is therefore not surprising that statisticians suggested using mixed models to analyze test data and determine “teacher effects.”
This is often explained to the public by analogy. One cannot accurately measure the quality of a teacher merely by looking at the scores on a single test at the end of a school year. If one teacher starts with all poorly prepared students, while another starts with all excellent, we would be misled by scores from a single test given to each class.

To account for such differences, we might use two tests, comparing scores from the end of one year to the next. The focus is on how much the scores increase rather than the scores themselves. That’s the basic idea behind “value added.” But value-added models (VAMs) are much more than merely comparing successive test scores.

Given many scores (say, grades 3–8) for many students with many teachers at many schools, one creates a mixed model for this complicated situation. The model is supposed to take into account all the factors that might influence test results — past history of the student, socioeconomic status, and so forth. The aim is to predict, based on all these past factors, the growth in test scores for students taught by a particular teacher. The actual change represents this more sophisticated “value added”— good when it’s larger than expected; bad when it’s smaller.

The best-known VAM, devised by William Sanders, is a mixed model (actually, several models), which is based on Henderson’s mixed-model equations, although mixed models originate much earlier [Sanders 1997]. One calculates (a huge computational effort!) the best linear unbiased predictors for the effects of teachers on scores. The precise details are unimportant here, but the process is similar to all mathematical modeling, with underlying assumptions and a number of choices in the model’s construction.

History
 
When value-added models were first conceived, even their most ardent supporters cautioned about their use [Sanders 1995, abstract]. They were a new tool that allowed us to make sense of mountains of data, using mathematics in the same way it was used to understand the growth of crops or the effects of a drug. But that tool was based on a statistical model, and inferences about individual teachers might not be valid, either because of faulty assumptions or because of normal (and expected) variation.

Such cautions were qualified, however, and one can see the roots of the modern embrace of VAMs in two juxtaposed quotes from William Sanders, the father of the value-added movement, which appeared in an article in Teacher Magazine in the year 2000. The article’s author reiterates the familiar cautions about VAMs, yet in the next paragraph seems to forget them:

Sanders has always said that scores for individual teachers should not be released publicly. “That would be totally inappropriate,” he says. “This is about trying to improve our schools, not embarrassing teachers. If their scores were made available, it would create chaos because most parents would be trying to get their kids into the same classroom.”
 
Still, Sanders says, it’s critical that ineffective teachers be identified. “The evidence is overwhelming,” he says, “that if any child catches two very weak teachers in a row, unless there is a major intervention, that kid never recovers from it. And that’s something that as a society we can’t ignore” [Hill 2000].
 
Over the past decade, such cautions about VAM slowly evaporated, especially in the popular press. A 2004 article in The School Administrator complains that there have not been ways to evaluate teachers in the past but excitedly touts value added as a solution:

“Fortunately, significant help is available in the form of a relatively new tool known as value-added assessment. Because value-added isolates the impact of instruction on student learning, it provides detailed information at the classroom level. Its rich diagnostic data can be used to improve teaching and student learning. It can be the basis for a needed improvement in the calculation of adequate yearly progress. In time, once teachers and administrators grow comfortable with its fairness, value-added also may serve as the foundation for an accountability system at the level of individual educators [Hershberg 2004, 1].”
 
And newspapers such as The Los Angeles Times get their hands on seven years of test scores for students in the L.A. schools and then publish a series of exposés about teachers, based on a value-added analysis of test data, which was performed under contract [Felch 2010]. The article explains its methodology:

“The Times used a statistical approach known as value-added analysis, which rates teachers based on their students’ progress on standardized tests from year to year. Each student’s performance is compared with his or her own in past years, which largely controls for outside influences often blamed for academic failure: poverty, prior learning and other factors.
 
Though controversial among teachers and others, the method has been increasingly embraced by education leaders and policymakers across the country, including the Obama administration.”

It goes on to draw many conclusions, including:

“Many of the factors commonly assumed to be important to teachers’ effectiveness were not. Although teachers are paid more for experience, education and training, none of this had much bearing on whether they improved their students’ performance.”
 
The writer adds the now-common dismissal of any concerns:

“No one suggests using value-added analysis as the sole measure of a teacher. Many experts recommend that it count for half or less of a teacher’s overall evaluation.
 
“Nevertheless, value-added analysis offers the closest thing available to an objective assessment of teachers. And it might help in resolving the greater mystery of what makes for effective teaching, and whether such skills can be taught.”

The article goes on to do exactly what it says “no one suggests” — it measures teachers solely on the basis of their value-added scores.

What Might Be Wrong with VAM?
 
As the popular press promoted value-added models with ever-increasing zeal, there was a parallel, much less visible scholarly conversation about the limitations of value-added models. In 2003 a book with the title Evaluating Value-Added Models for Teacher Accountability laid out some of the problems and concluded:

“The research base is currently insufficient to support the use of VAM for high-stakes decisions. We have identified numerous possible sources of error in teacher effects and any attempt to use VAM estimates for high-stakes decisions must be informed by an understanding of these potential errors [McCaffrey 2003, xx].”
 
In the next few years, a number of scholarly papers and reports raising concerns were published, including papers with such titles as “The Promise and Peril of Using Valued-Added Modeling to Measure Teacher Effectiveness” [RAND, 2004], “Re-Examining the Role of Teacher Quality in the Educational Production Function” [Koedel 2007], and “Methodological Concerns about the Education Value-Added Assessment System” [Amrein-Beardsley 2008].

What were the concerns in these papers? Here is a sample that hints at the complexity of issues.
• In the real world of schools, data is frequently missing or corrupt. What if students are missing past test data? What if past data was recorded incorrectly (not rare in schools)? What if students transferred into the school from outside the system?

• The modern classroom is more variable than people imagine. What if students are team-taught? How do you apportion credit or blame among various teachers? Do teachers in one class (say mathematics) affect the learning in another (say science)?

• Every mathematical model in sociology has to make rules, and they sometimes seem arbitrary. For example, what if students move into a class during the year? (Rule: Include them if they are in class for 150 or more days.) What if we only have a couple years of test data, or possibly more than five years? (Rule: The range three to five years is fixed for all models.) What’s the rationale for these kinds of rules?

• Class sizes differ in modern schools, and the nature of the model means there will be more variability for small classes. (Think of a class of one student.) Adjusting for this will necessarily drive teacher effects for small classes toward the mean. How does one adjust sensibly?

• While the basic idea underlying value-added models is the same, there are in fact many models. Do different models applied to the same data sets produce the same results? Are value-added models “robust”?

•Since models are applied to longitudinal data sequentially, it is essential to ask whether the results are consistent year to year. Are the computed teacher effects comparable over successive years for individual teachers? Are value-added models “consistent”?

These last two points were raised in a research paper [Lockwood 2007] and a recent policy brief from the Economic Policy Institute, “Problems with the Use of Student Test Scores to Evaluate Teachers”, which summarizes many of the open questions about VAM:

“For a variety of reasons, analyses of VAM results have led researchers to doubt whether the methodology can accurately identify more and less effective teachers. VAM estimates have proven to be unstable across statistical models, years, and classes that teachers teach. One study found that across five large urban districts, among teachers who were ranked in the top 20% of effectiveness in the first year, fewer than a third were in that top group the next year, and another third moved all the way down to the bottom 40%. Another found that teachers’ effectiveness ratings in one year could only predict from 4% to 16% of the variation in such ratings in the following year. 
 
“Thus, a teacher who appears to be very ineffective in one year might have a dramatically different result the following year. The same dramatic fluctuations were found for teachers ranked at the bottom in the first year of analysis. This runs counter to most people’s notions that the true quality of a teacher is likely to change very little over time and raises questions about whether what is measured is largely a “teacher effect” or the effect of a wide variety of other factors [Baker 2010, 1].”
 
In addition to checking robustness and stability of a mathematical model, one needs to check validity. Are those teachers identified as superior (or inferior) by value-added models actually superior (or inferior)? This is perhaps the shakiest part of VAM. There has been surprisingly little effort to compare valued-added rankings to other measures of teacher quality, and to the extent that informal comparisons are made (as in the LA Times article), they sometimes don’t agree with common sense.
None of this means that value-added models are worthless—they are not. But like all mathematical models, they need to be used with care and a full understanding of their limitations.

How Is VAM Used?
 
Many studies by reputable scholarly groups call for caution in using VAMs for high-stakes decisions about teachers.

A RAND research report: The estimates from VAM modeling of achievement will often be too imprecise to support some of the desired inferences [McCaffrey 2004, 96].

A policy paper from the Educational Testing Service’s Policy Information Center: VAM results should not serve as the sole or principal basis for making consequential decisions about teachers. There are many pitfalls to making causal attributions of teacher effectiveness on the basis of the kinds of data available from typical school districts. We still lack sufficient understanding of how seriously the different technical problems threaten the validity of such interpretations [Braun 2005, 17].

A report from a workshop of the National Academy of Education: Value-added methods involve complex statistical models applied to test data of varying quality. Accordingly, there are many technical challenges to ascertaining the degree to which the output of these models provides the desired estimates [Braun 2010].

And yet here is the LA Times , publishing value-added scores for individual teachers by name and bragging that even teachers who were considered first-rate turn out to be “at the bottom”. In an episode reminiscent of the Cultural Revolution, the LA Times reporters confront a teacher who “was surprised and disappointed by her [value-added] results, adding that her students did well on periodic assessments and that parents seemed well-satisfied” [Felch 2010]. The teacher is made to think about why she did poorly and eventually, with the reporter’s help, she understands that she fails to challenge her students sufficiently. In spite of parents describing her as “amazing” and the principal calling her one of the “most effective” teachers in the school, she will have to change. She recants: “If my student test scores show I’m an ineffective teacher, I’d like to know what contributes to it. What do I need to do to bring my average up?”

Making policy decisions on the basis of value-added models has the potential to do even more harm than browbeating teachers. If we decide whether alternative certification is better than regular certification, whether nationally board certified teachers are better than randomly selected ones, whether small schools are better than large, or whether a new curriculum is better than an old by using a flawed measure of success, we almost surely will end up making bad decisions that affect education for decades to come.

This is insidious because, while people debate the use of value-added scores to judge teachers, almost no one questions the use of test scores and value-added models to judge policy. Even people who point out the limitations of VAM appear to be willing to use “student achievement” in the form of value-added scores to make such judgments. People recognize that tests are an imperfect measure of educational success, but when sophisticated mathematics is applied, they believe the imperfections go away by some mathematical magic. But this is not magic. What really happens is that the mathematics is used to disguise the problems and intimidate people into ignoring them—a modern, mathematical version of the Emperor’s New Clothes.

What Should Mathematicians Do?
 
The concerns raised about value-added models ought to give everyone pause, and ordinarily they would lead to a thoughtful conversation about the proper use of VAM. Unfortunately, VAM proponents and politicians have framed the discussion as a battle between teacher unions and the public.

Shouldn’t teachers be accountable? Shouldn’t we rid ourselves of those who are incompetent? Shouldn’t we put our students first and stop worrying about teacher sensibilities? And most importantly, shouldn’t we be driven by the data?

This line of reasoning is illustrated by a recent fatuous report from the Brookings Institute, “Evaluating Teachers: The Important Role of Value-Added” [Glazerman 2010], which dismisses the many cautions found in all the papers mentioned above, not by refuting them but by asserting their unimportance. The authors of the Brookings paper agree that value-added scores of teachers are unstable (that is, not highly correlated year to year) but go on to assert:

“The use of imprecise measures to make high-stakes decisions that place societal or institutional interests above those of individuals is widespread and accepted in fields outside of teaching [Glazerman 2010, 7].”
 
To illustrate this point, they use examples such as the correlation of SAT scores with college success or the year-by-year correlation of leaders in real estate sales. They conclude that “a performance measure needs to be good, not perfect”. (And as usual, on page 11 they caution not to use value-added measures alone when making decisions, while on page 9 they advocate doing precisely that.)

Why must we use value-added even with its imperfections? Aside from making the unsupported claim (in the very last sentence) that “it predicts more about what students will learn ... than any other source of information,” the only apparent reason for its superiority is that value-added is based on data. Here is mathematical intimidation in its purest form—in this case, in the hands of economists, sociologists, and education policy experts.

Of course we should hold teachers accountable, but this does not mean we have to pretend that mathematical models can do something they cannot. Of course we should rid our schools of incompetent teachers, but value-added models are an exceedingly blunt tool for this purpose. In any case, we ought to expect more from our teachers than what value-added attempts to measure.

A number of people and organizations are seeking better ways to evaluate teacher performance in new ways that focus on measuring much more than test scores. (See, for example, the Measures of Effective Teaching project run by the Gates Foundation.) Shouldn’t we try to measure long-term student achievement, not merely short-term gains? Shouldn’t we focus on how well students are prepared to learn in the future, not merely what they learned in the past year? Shouldn’t we try to distinguish teachers who inspire their students, not merely the ones who are competent?

When we accept value-added as an “imperfect” substitute for all these things because it is conveniently at hand, we are not raising our expectations of teachers, we are lowering them. And if we drive away the best teachers by using a flawed process, are we really putting our students first?

Whether naïfs or experts, mathematicians need to confront people who misuse their subject to intimidate others into accepting conclusions simply because they are based on some mathematics. Unlike many policy makers, mathematicians are notbamboozled by the theory behind VAM, and they need to speak out forcefully. Mathematical models have limitations. They do not by themselves convey authority for their conclusions. They are tools, not magic. And using the mathematics to intimidate — to preempt debate about the goals of education and measures of success — is harmful not only to education but to mathematics itself.

References
 
Audrey Amrein-Beardsley, Methodological concerns about the education value-added assessment system, Educational Researcher 37 (2008), 65–75. http:// dx.doi.org/10.3102/0013189X08316420
Eva L. Baker, Paul E. Barton, Linda Darling-Hammond, Edward Haertel, Hellen F. Ladd, Robert L. Linn, Diane Ravitch, Richard Rothstein, Richard J. Shavelson, and Lorrie A. Shepard, Problems with the Use of Student Test Scores to Evaluate Teachers, Economic Policy Institute Briefing Paper#278, August 29, 2010, Washington, DC. http://www.epi.org/publications/entry/bp278
Henry Braun, Using Student Progress to Evaluate Teachers:
A Primer on Value-Added Models, Educational Testing Service Policy Perspective, Princeton, NJ, 2005. http://www.ets.org/Media/Research/pdf/PICVAM.pdf
Henry Braun, Naomi Chudowsky, and Judith Koenig, eds., Getting Value Out of Value-Added: Report of a Workshop, Committee on Value-Added Methodology for Instructional Improvement, Program Evaluation, and Accountability; National Research Council, Washington, DC, 2010.
http://www.nap.edu/catalog/12820.html
Donald T. Campbell, Assessing the Impact of Planned Social Change, Dartmouth College, Occasional Paper Series, #8, 1976. http://www.eric.ed.gov/PDFS/ED303512.pdf
Jason Felch, Jason Song, and Doug Smith, Who’s teaching L.A.’s kids?, Los Angeles Times, August 14, 2010. http://www.latimes.com/news/local/la-me-teachers-value-20100815,0,2695044.story
Trip Gabriel, Under pressure, teachers tamper with tests, New York Times, June 11, 2010.
http://www.nytimes.com/2010/06/11/education/11cheat.html
Steven Glazerman, Susanna Loeb, Dan Goldhaber, Douglas Staiger, Stephen Raudenbush, Grover Whitehurst, Evaluating Teachers: The Important Role of Value-Added, Brown Center on Education Policy at Brookings, 2010. http://www.brookings.edu/reports/2010/1117_evaluating_teachers.aspx
Ted Hershberg, Virginia Adams Simon and Barbara Lea Kruger, The revelations of value-added: An assessment model that measures student growth in ways that NCLB fails to do, The School Administrator, December 2004.
http://www.aasa.org/SchoolAdministratorArticle.aspx?id=9466
David Hill, He’s got your number, Teacher Magazine,
May 2000 11(8), 42–47.
http://www.edweek.org/tm/articles/2000/05/01/08sanders.h11.html
Cory Koedel and Julian R. Betts, Re-Examining the Role of Teacher Quality in the Educational Production Function, Working Paper #2007-03, National Center on Performance Initiatives, Nashville, TN, 2007.
http://economics.missouri.edu/working-papers/2007/wp0708_koedel.pdf
Daniel Koretz, Measuring Up: What Educational Testing Really Tells Us , Harvard University Press, Cambridge, Massachusetts, 2008.
E. F. Lindquist, Preliminary considerations in objective test construction, in Educational Measurement (E. F. Lindquist, ed.), American Council on Education, Washington DC, 1951.
J. R. Lockwood, Daniel McCaffrey, Laura S. Hamilton, Brian Stetcher, Vi-Nhuan Le, and Felipe Martinez, The sensitivity of value-added teacher effect estimates to different mathematics achievement measures, Journal of Educational Measurement 44(1) (2007), 47–67. http://dx.doi.org/10.1111/j.1745-3984.2007.00026.x
Daniel F. McCaffrey, Daniel Koretz, J. R. Lockwood, and Laura S. Hamilton, Evaluating Value-Added Models for Teacher Accountability, RAND Corporation, Santa Monica, CA, 2003.
http://www.rand.org/pubs/monographs/2004/RAND_MG158.pdf
Daniel F. McCaffrey, J. R. Lockwood, Daniel Koretz, Thomas A. Louis, and Laura Hamilton, Models for value-added modeling of teacher effects, Journal of Educational and Behavioral Statistics 29(1), Spring 2004, 67-101.
http://www.rand.org/pubs/reprints/2005/RAND_RP1165.pdf
RAND Research Brief, The Promise and Peril of Using Value-Added Modeling to Measure Teacher Effectiveness, Santa Monica, CA, 2004. http://www.rand.org/pubs/research_briefs/RB9050/RAND_RB9050.pdf
William L. Sanders and Sandra P. Horn, Educational Assessment Reassessed: The Usefulness of Standardized and Alternative Measures of Student Achievement as Indicators of the Assessment of Educational Outcomes, Education Policy Analysis Archives, March 3(6) (1995). http://epaa.asu.edu/ojs/article/view/649
W. Sanders, A. Saxton, and B. Horn, The Tennessee value-added assessment system: A quantitative outcomes-based approach to educational assessment, in Grading Teachers, Grading Schools: Is Student Achievement a Valid Evaluational Measure? (J. Millman, ed.), Corwin Press, Inc., Thousand Oaks, CA, 1997, pp 137–162.
-0-
Follow The Answer Sheet every day by bookmarking http://www.washingtonpost.com/blogs/answer-sheet. And for admissions advice, college news and links to campus papers, please check out our Higher Education page. Bookmark it!

Monday, February 13, 2012

Central Texas districts vary on handling of new end-of-course exams

An important board meeting going down tonight. What parents and community need to remember is that the 15% rule is just one element of the larger system. There is no research that shows that this will facilitate greater learning and college readiness. In fact, universities and entire systems (e.g., University of California) have done away from relying on standardized assessment performance in admissions decisions.

What proponents of the 15% rule (which you can count on one hand) are actually doing is contaminating the most significant research-based indicator that we know helps to predict how students will perform in college: GPA.

-Patricia


Austin American-Statesman
February 13, 2012

Even before the state education commissioner began questioning Texas' emphasis on testing and the effect on teaching, districts had started backing off plans that would made the state's new standardized test count toward students' grade-point averages and class rank.

Now, officials in Austin, Manor and Pflugerville said they will take another look at policies on how to incorporate the new State of Texas Assessments of Academic Readiness into a student's final grade. Ninth-graders start taking the end-of-course exams next month.

The STAAR is replacing the Texas Assessment of Knowledge and Skills, and districts across the state are grappling with how to handle the new rules. Lawmakers have required the STAAR to count as 15 percent of the student's grade in each course, unlike the TAKS, but did not spell out how to do so. Instead, that has been left up to individual districts.

Some, including the Eanes and Round Rock districts, have already said they will calculate the STAAR as part of the final grade that appears for the course on a student's transcript but said that final grade won't be used in calculating a student's GPA or class rank, which is what colleges will focus on.

In Texas, being among the top of the class guarantees admission to all state universities.

On Friday, Pflugerville officials said they may make changes to previously approved policies that called for including the exams in factoring GPAs and class rank.

Manor Superintendent Andrew Kim, in a reversal from his recent testimony before state lawmakers during a hearing on the exams, said this week that he also will reconsider.

"Originally, we had been somewhat strong about adding it in," Kim said. "But after looking at the tenor of what's out there ... and with this all possibly being reconsidered in the next legislative session, we figure it would be easier to (leave it out now and) add it in later on."

The Austin district has polled parents on four possible ways to include the STAAR in student grades. All the options would affect class rank. But under pressure from concerned parents — and after noting what neighboring districts have done — several Austin school board members have said they'd also like to reconsider. Austin trustees plan to discuss the issue at a meeting tonight.

Critics say school districts are taking a risk in trying to get around a new state law that clearly intends to strengthen school accountability.

"When a high school sends a transcript to a college, they recalculate the GPA," said Drew Scheberle, senior vice president for education and talent at the Austin Chamber of Commerce, one of several business organizations that have been standing by the switch to higher standards based on college readiness.

"So while Westlake is trying to be cute about this, what they could end up doing is (they could) give the wrong impression to their students," that the tests don't matter, Scheberle said.

The districts are wrestling with the issue as state education Commissioner Robert Scott indicated he now thinks testing has gone too far.

At a Texas Association of School Administrators meeting in January, he said that the state testing system has become a "perversion of its original intent" and that he was looking forward to "reeling it back in."

As for how the districts decide to factor the test into student grades, he said the call is not his to make.

"State law is very clear," Scott said in a statement. "I do not have the authority to issue a uniform grading policy for Texas public schools. Our education system is based on a presumption of local control by elected school boards. In short, my office has been given no authority to override the decisions of local school boards on their grading policies and the applications of the end-of-course exams to a student's grade. An attempt to do so would only invite litigation that would cause further confusion.

"The law creating the 15 percent grading policy requirement was passed by the Texas Legislature in 2007. Two additional regular sessions have occurred since then, and lawmakers have not changed the grading policy provisions. I understand the situation may be causing confusion, and I look forward to working with lawmakers to sort this out in the next legislative session," he said.

Of particular interest here to Scheberle and others who support the changes is where districts are setting minimum scores for the end-of-course exams. Austin's proposals vary, from 50 to 69.

The lowest grade Georgetown students would get would be a 69. Pflugerville has set the minimum score at 60.

"A student should not be able to hand in a blank test and get a 60 or above," Scheberle said. "Regardless of the law, it demoralizes the other students who are working hard, who do try, and it's not fair to those students. It's cynical and against the aims of education, which is to learn."

Pflugerville officials said the process has been difficult for administrators to explain to parents.

"We're responding to I think a lack of information," said Deputy Superintendent Keith McBurnett.

He added that the whole idea of tying the assessment to students' grades perhaps needs to be delayed, if not reconsidered. There's no indication the state would do so.

"The assessment is high stakes enough. It counts toward graduation," McBurnett said. "It seems to me as I've watched the dripping out of information, the best solution would have been to at least delay this another year and have all the information in place."

While Scheberle disagrees the test should not be tied to grades, he said he does think Scott could have done a better job communicating with districts and parents about the new tests and grading requirements and preparing for the more rigorous exam.

"He had 5½ years to prepare and he's not ready, so now he's blaming it on other forces," he said.

Scott declined an interview with the American-Statesman.

Scheberle said the purpose of the accountability system is to make sure students are ready for college and career, adding that the system currently doesn't do enough to meet the demands for highly qualified employees in Central Texas.

"We have 55,000 unemployed here, and many of them have less than an associate's degree," he said. "It's hard for us to grow jobs if we don't have college-ready graduates, and this commissioner being squishy (on improving college readiness) isn't helping the situation."

Kim, the Manor superintendent, said he supports Scott's comments about testing being taken too far and feels that there are other ways for students to demonstrate knowledge. Manor has started several classroom initiatives in which students demonstrate learning through projects.

"I am a very big proponent and supporter of accountability and how it helps us improve education, but I do feel there are alternative ways to assess rather than just the one way we are looking at now," he said. "This whole debate has shown how there is this uniqueness to having local control.

"I think the issue now is for the first time the state has kind of created a bridge toward state assessment and local control of grades, so what does that bridge look like? We feel that if the bridge does exist, then we want to do what is the very best for our kids in terms of motivating them to do well."

lheinauer@statesman.com; 445-3694
View as multiple pages

Central Texas districts vary on handling of new end-of-course exams

The State of Texas Assessments of Academic Readiness is replacing the Texas Assessment of Knowledge and Skills. Lawmakers have required the STAAR to count as 15 percent of a student's grade in each course but did not spell out how to do so. Instead, that has been left up to individual districts.

District Affects GPA? Affects class rank? Minimum score Notes

Austin Pending Pending Pending Asking parents to choose among four options that would set minimum scores

between 50 and 69. Final decision expected by March.

Del Valle Yes Yes Pending Officials seeking public input on how to count exams toward grades.

Dripping Springs Yes Yes Pending Officials said no further information was available.

Eanes No No 67 Exam grade will appear on transcripts.

Georgetown Yes Yes 69

Hays Pending Pending Pending Officials are reviewing options and hope to have a decision in March.

Hutto Yes Yes Pending Officials are deciding between a minimum score between 0 and 60.

Lake Travis No No Pending A final decision will be made after spring break.

Leander Pending Pending Pending Administrators are recommending that the exams not affect GPA or class rank.

Manor Pending Pending Pending District is looking at setting a bottom score between 60 and 69.

Round Rock No No 60 Students will get separate ‘class grades' and final grades, so that GPA and

class rank will not be affected.

Pflugerville Pending Pending 60 Administrators are waiting for more information from the state on passing standards.

San Marcos Pending Pending Pending Administrators are waiting for more information from the state on passing standards.

Source: Area school districts. Not all responded to requests for information.

Meeting today

The Austin school board will meet to discuss end-of-course exams and naming a new early childhood development center, among other things. 6:30 p.m., Carruth Administration Center Board Auditorium, Room B100, 1111 W. Sixth St.

Sunday, February 12, 2012

Sticking With GPA is Smarter Than Make or Break

O. Ricardo Pimentel, Express-News columnist

Stay tuned, everybody, to Patricia Lopez' dissertation research on HB3 that lays bare the who, what and how behind the new STAAR system of accountability and testing.  Her work builds on the work of the Texas Center for Education Policy at UT-Austin, as well as earlier efforts that  began over a decade ago in tandem with key community players like NAACP, LULAC, MASBA, the GI Forum, MALDEF (that spearheaded the federal trial [by Atty. Al Kauffman]), numerous scholars—including Drs. Linda McNeil, Richard Valencia, Kris Sloan, Gary Orfield and myself and other individuals and groups with challenges to TAKS and high-stakes accountability.  It has been a long, hard-fought struggle but fortunately, we were able to eliminate high-stakes testing for third-graders in the state of Texas.  This latest iteration, among other things, consists of 12 end-of-course exams, 45 testing days, and a contaminating of GPA (that this piece covers really well), by all trustworthy accounts, is simply headed in the wrong direction.  You will not find any national professional organization of any repute that supports high-stakes testing--regardless of the fact that all states have them now for schools as a result of NCLB.  Even the testing companies themselves say that tests should never be used in the way that they have been, and are continuing to be, used. Texans need to insist on research-based, fair and equitable approaches to student assessment and evaluation  Much of this has been documented for quite sometime in this blog. There is a search function at the bottom of the page that will give you access to information over the years, from legislative session to legislative session.

So yes, folks.  More to come.

Angela

The goal of the new high-stakes STAAR testing that Texas will launch in the spring is purportedly to increase the college readiness of the state's students.
Unfortunately, it's entirely likely that this testing will diminish what is the single best indicator. That would be grade point average.
With the State of Texas Assessment of Academic Readiness, high school students and their parents are supposed to be comforted by the removal of a single high-stakes test that determines a diploma.
But in its place will be 12 end-of-course (EOC) tests — still high stakes, but in 12 installments. Fail to reach a minimum score on a single EOC and it doesn't count toward the cumulative scores of all EOCs. Don't reach this cumulative and you don't graduate. Fail to reach the minimum passing score for some classes and you don't get a “recommended” or “distinguished” high school diploma.
In furtherance of a noble goal, college readiness, we're embarking on a testing arms race that will require more testing days and a spreadsheet to know if you're on track or not. But why? The best predictor of college success is just plain ol' high school grade point average.
“High school grades are sometimes viewed as the least reliable indicator than standardized tests because grading standards differ across schools,” says a 2009 paper. “Yet ... grades still outperform standardized tests in predicting college outcomes.”
Writing this were Richard C. Atkinson and Saul Geiser. Atkinson is a former president of the University of California system and Geiser is a respected research associate at the Center for Studies in Higher Education at the University of California, Berkeley.
They were talking mostly about SATs and ACTs. They wrote that standards-based assessments in high schools hold promise but said even these have had problems as predictors. They point to GPA as still most determinant.
The potential with STAAR's 12 end-of-course tests through three high school years is that GPA becomes “contaminated.”
This is a word used both by Patricia Lopez, whose doctoral dissertation at the University of Texas is on STAAR, and Angela Valenzuela, who has studied high stakes testing as a UT professor and director of the Texas Center for Education Policy. Lopez is also a research associate at the center.
They explain that these EOCs will count for 15 percent of grades in those core courses. And what is tested is also a problem.
“What is the test really telling you if 65 percent of the test is focused on 30 percent of the curriculum?” asked Lopez.
Lopez also noted that this potentially undercuts the state's 10 percent rule — dictating that the top 10 percent in high schools qualify for admission to the state's top public universities.
Why then is Texas doing this (and, by the way, without adequate funding, either for remediation or testing)?
Let's just call it the triumph of what sounds good — high standards and the teeth to enforce them — over what is actually good.
Needed here: The realization that many factors determine college success — GPA key among them, successful course completion counting for much but testing, perhaps, counting for least of all.

o.ricardo.pimentel@express-news.net

NCLB Waivers: Implications for Testing, Standards Implementation

The big education news out of Washington yesterday was that 10 states have been awarded waivers of key accountability provisions in the No Child Left Behind Act. This action by the U.S. Department of Education holds some potentially important implications for teaching and learning, as it essentially opens the door for states to rethink their priorities and approaches in evaluating schools (and districts). And accountability systems create a lot of pressure to influence what gets taught—and how—in the classroom.


Read on here.

Christie Praises 'No Child Left Behind' Waiver for NJ

The governor throws his support behind the Obama administration's decision to give New Jersey a pass on No Child Left Behind regulations.

Gov. Chris Christie praised the federal government's decision this week to grant a waiver releasing New Jersey from the rules of the No Child Left Behind (NCLB) Act.
On Thursday, it was announced 10 states, including New Jersey, would be granted waivers by the federal government that would allow them to bypass the rules and regulations of NCLB and give them greater freedom in developing systems of student accountability.
The waiver comes with an approval of the state's required annual NCLB application, in which state officials advocated for the existing standardized testing benchmarks to be abolished. A new form of measuring student progress is to begin in September of this year.
As part of New Jersey's successful waiver application, the Christie administration outlined plans for three principles that are in line with the goals of the Obama administration, according to the governor's office.
These principles include college- and career-ready expectations for all students, state-developed differentiated recognition, accountability and support, and providing support for effective instruction and leadership.
"The Obama administration's approval of our education reform agenda contained in this application confirms that our bold, common sense, and bipartisan reforms are right for New Jersey and shared by the president and (Education) Secretary (Arne) Duncan's educational vision for the country," Christie said in a prepared statement. "This is not about Democrats or Republicans—it is about pursuing an agenda in the best interest of our children whose educational needs are not being met, and those who are getting a decent education but deserve a great one." 
Through NCLB, student groups were measured on their separate performances, and schools were classified as making Adequate Yearly Progress (AYP) or "In Need of Improvement."
Regardless of socioeconomic background, race or individual ability, all students were expected to meet the benchmarks set by the tests. If one group of students did not score high enough, an entire school could be classified as "In Need of Improvement."
Schools that repeatedly failed to make AYP faced penalties that could include anything from reduced funding, firing teachers and administrative staff, offering more tutoring and support services, and in extreme cases, closing, or laying off staff.
NCLB was signed into law in 2001 by then-president George W. Bush. It had a 2014 goal, in which all students were to be proficient in math and reading.
Schools in New Jersey will no longer have to meet NCLB benchmarks, but instead will be subject to a "fairer and more nuanced accountability system ... that measures schools based on both growth and absolute attainment," a release from the governor's office said.
This new system will separate schools into three tiers: "Priority Schools," which will be chosen from the lowest performing 5 percent of Title I schools statewide; "Focus Schools," which will be chosen from at least 10 percent of Title I schools; and "Reward Schools," or those schools that demonstrate high student performance, or are making progress in closing achievement gaps between student groups, according to the release.
These schools will be identified during the summer, with interventions to begin during the 2012-13 school year.
Randi Weingarten, president of the American Federation of Teachers, the largest education union in the country, said the Obama administration was reacting to the calls from parents and educators to fix what is often considered an imperfect system, but she was cautious in her appraisal of the situation.
If state officials do not work with teachers and administrators to design their own new system, which should center around student learning and not test scores, NCLB could be replaced by a system that is either no different, or worse than the current system, she said.
"We remain concerned that some states may use these waivers to simply put metrics on top of poorly constructed and implemented evaluation systems," she said.
Related Topics: Gov. Chris Christie, No Child Left Behind, and No Child Left behind waiver

Saturday, February 11, 2012

Announcing: Energy of a Nation Curriculum

Proudly presenting...

The Advocates for Human Rights proudly announces the publication of Energy of a Nation: Immigrants in America, 3rd Edition. This curriculum is a distinctive, comprehensive guide to teaching students about immigration in the United States with human rights analysis woven throughout lessons. It can be used in several disciplines and with different age levels, and it is available free for download on The Advocates' EnergyofaNation.org website.


Energy of a Nation, 3rd Ed. provides learners with fundamental concepts and critical context, such as:

* Definitions of key immigration-related terms;

* Admission categories and processes;

* Statistics and trends of immigrants over time;

* Root causes of undocumented immigration;

* The complex realities of removal through the immigration courts;

* Other countries' experience with, and response to, immigration;

* Nativism and public discourse around immigration;

* Push and pull factors that cause people to move;

* The special case of refugees and asylum seekers;

* The human rights of immigrants;

* Local and national U.S. policy considerations; and

* Service learning opportunities to create a welcoming school and community.

The curriculum is filled with engaging, student-centered activities that follow best practices for human rights education (HRE). Students learn by exploring their own immigrant history; role-playing a refugee's journey; deciding under what conditions they might risk being undocumented; playing games to understand the immigration system; drawing representative pictures of policies; rehearsing deliberative dialogue; constructing a gallery of nativism over the centuries; and creating a service learning project for their classroom or school.

Using the HRE framework for immigration allows students to acquire the knowledge to understand immigration topics, but also to gain the skills and values necessary to process future information or experiences. Students learn to put information in context, check it against reliable sources, consider root causes, make essential connections, and participate in democratic processes. They are provided the opportunity to view themselves and the United States as actors in a global, fluid movement of people and to see the human beings that make up these mass flows as individuals - each with a story, a life, and the same rights that bind us all.

The entire curriculum will be available to school and community educators online, free of charge. In addition, staff will respond to teacher requests to demonstrate lessons in K-12 classrooms and provide trainings to teachers and other school staff on immigration and human rights through professional development workshops.

Idiotas: Enduring Fools, by Rodolfo F. Acuña


Idiotas
Enduring Fools
By
Rodolfo F. Acuña


Growing up, we used a plethora of words to dismiss fools. Fom two different worlds, my father was from Jalisco so his sayings were always blander my mother’s Sonorense expressions, which always seemed franker and more to the point.  If you were ugly, they called you el feo.  It was the cow culture that reveled in a no bull sh.. mentality. When I messed up badly, I was the pinche güero or the pinche buey, which depending on how it was said was generally a put down.

Words such as cabrón or chingado were rarely used unless in anger and mostly directed at someone outside the family.  Even to this day they are words that are not taken lightly in Mexico, especially if used in the context of chinga tu madre.

(It was unlike today when an 85 year old lady will flip you off on the freeway.) 

When you thought someone was stupid and just did not want anything to with them, it was estúpido, imbécil, baboso or pendejo. They had a shock value.  My family was more passive and would just give you the señal de la cruz  -- why call them names if they don’t exist for you.

In the past several years I have found myself giving most Tucson racists the sign of the cross – they are brain-dead. 

Under normal circumstances, I would have given people like Arizona Attorney Tom Horne, Superintendent of Public Instruction John Huppenthal and the great majority of the board members of the Tucson Unified Schools the señal de la cruz.  However, this was impossible because they do so much damage to an entire community. Reaching back into my consciousness I came up with the perfect descriptive word, idiota, a word that always had a poetic sound. Pinche güero was nice next to idiota!


It was a word that had to be said in Spanish. The word sounded dull in English.  Idiot just doesn’t cut it. It is as if it is spoken by Joe Pesci. Idiota has the ring of Sophia Loren strutting down the streets of Pozuolli, Italy, waving her finger at Marcelo Mastroianni and shouting idiota. It sounds like an angry Luciano Pavarotti aura. It is not mealy mouthed like idiot.

Arizona Governor Jan Brewer wagging her finger at President Obama recently caught my attention; it a gesture that betrayed her IQ and upbringing.  It was a punk act, and she did it because she knew she could get away with it.

Can you imagine her wagging her finger in John F. Kennedy’s face; JFK had enough Boston Irish not to have gotten angry, but he would have gotten even.  Kennedy was just too rich to mess with, and father Joe Kennedy and brother Bobby Kennedy would have taken the insult personally

Lyndon B. Johnson would have told her where to stick it, and Richard Nixon would have fetched J Edgar Hoover on her.

Hilary Clinton would have taken her outside.  She never would have tried that cheap trick with Congresswoman Maxime Waters who would have had her for breakfast. Brewer is an idiota and a coward who she did it for show. Like many of her supporters she has no class.

It goes without saying that Horne and Huppenthal are idiotas.  Every time Horne says that he is with Martin Luther King and wants to judge a person by the content of his character rather than the color of his skin, he desecrates King’s legacy.

Horne judged La Raza Studies without knowing the definition of La Raza, and condemned the whole program without making a single classroom visit.  Although he invokes King, Horne attends neo-Nazi gatherings like "Arizona Mainstream Project," that promotes Glenn Beck and Cleon Skousen.  Horne who avoided military invokes phrases such as MAS “must be destroyed.”  Yet, he is afraid to visit a MAS class.

Huppenthal dismissed a costly audit because he had heard differently from unspecified sources.  Huppenthal compared TUSD Mexican students to the Hitler Nazi Jugend paramilitary organization at a Pima County Republican luncheon.

Tucson has more than its share of idiotas. The truth be told, TUSD superintendent John Pedicone banned Shakespeare’s The Tempest because the racist Prospero reminded him too much of himself. When he was campaigning for his position, he voiced support of the MAS program saying that he heard good things about it. Once he had his $300,000 job in tow, he became the bagman for the Southern Arizona Leadership Council – a gaggle of modern day Robber Barons.

Board President Mark Stegeman, who has not shown an ounce of intellectual curiosity,
has allowed Minutemen to speak while suppressing the Mexican American community. Stegemam is supposed to be evaluating curriculum but has come up with statements like La Raza is a cult because they use the farmworker handclap.

Like his Tea Party colleague Michael Hicks, he is continuously bested by students who have to correct him.

Alexandre Sugiyama, the latest appointment to the board, just sits there and votes with the majority that is rounded off by Miguel Cuevas, a college student who is a wannabe mover and shaker; he thinks that by siding with the SALC puppets, he’ll turn white. Cuevas would be the perfect Pancho in the Cisco Kid, or better still, the Lone Ranger’s sidekick Tonto.

However, it is too easy to make fun of sick people. In doing so, you can cross the line and become like them.  It is not good taste to bait a mad dog. But, I have to confess that I have crossed over this line.

Aside from the main characters, the bagmen, there are scores of other idiotas. You have “Fringe characters” such Jared Laughner who shot Congresswoman Gabrielle Giffords (D-Ariz.), and killed six others. The wannabes can be found on the pages of the Tucson Citizen or other Arizona newspaper, grabbing their cheap thrills by writing commentaries to opinion pieces.

They don’t write anything of substance, coming up with inane comments such as “go back to Mexico” or “this America.” Mostly they are anonymous like the flasher they get momentary thrills.

In one case it was a CIA operative in Guatemala whose wife teaches in the Spanish Department at the University of Arizona who keeps up with Tucson via the internet and likes to feel important by dropping what he considers insider information.

There is little dialogue going on, just everyone trying to one up each other.   Recently I wrote on the Spanish Inquisition and a person with the pseudonym of Marcus Tullius commented,  “… of course, relate so well to the Spanish Inquisition because the inquisitors like Torquemada are your political antecedents…”  He completely ignored the point of the article that likened the destruction of Mexican American Studies and the banning of the books to the Inquisition. In another commentary he wrote, “HA HA.  Really?  I mean REALLY?  You’re hilarious!  Bro, you had to sue for tenure when you were publishing in crap journals.  Peer review is great when your peers aren’t first graders.  I desk reject your crappy comments.  No revise and resubmit for you!”
Marcus Tullius is of course Marcus Tullius Cicero purportedly Rome's greatest speaker and writer that greatly influenced European thought. Having read Cicero in Latin during my junior year in high school, I can say Mr. Anonymous is no Cicero.

This pedant assumes that if he adopts Cicero’s name that makes him famous.

Just as comical is that many of his cohorts seem to think that Marcus Tullius is Greek. Unfortunately, with people who hide under the cover of anonymity there can be no education, and they will forever remain idiotas.  So, in the name of the father, the son and the Holy Ghost, “Liberate te ex inferis.”

                

Joe Brewer: An interdisciplinary approach to understanding complex issues

This is a very thought-provoking interview with Joe Brewer, a colleague of George Lakoff and how paradigm shifts come about. 


I like his statement about diversity:  "Biodiversity is actually very healthy for sustainability because one thing we need is resilience. We need to be able to adapt to changing circumstances and if all of the cultures of the world are too similar to each other and if the way they all align is not healthy, then we are more at risk, and global civilisation could collapse completely.  When we talk about cultural change driving change in economics and politics, then we can find the strength of culture at different places and bring them together and drive innovation by plugging into places where cultures come into contact with each other."


Very interesting read.


-Angela


Joe Brewer: An interdisciplinary approach to understanding complex issues

By Bhavani Prakash

Joe Brewer
Seattle based JOE BREWER is one amongst a rare and emerging breed of interdisciplinary experts around the world. In an era of specialisations, what’s missing is a holistic view that cuts across various discipines, whether it comes to addressing climate change or societal change.  Brewer steps in with his unique perspectives on ‘cognitive policy.’ As Founder and Director of Cognitive Policy Works, he paints a sweeping and fascinating canvas covering the human mind, human behaviour, public policy, social media and societal change. 

EWTT: Tell us about how you came to be an interdisciplinary expert?

Joe Brewer: Even from a very young age I was fascinated by two things. One was patterns in the world, and the other was people.  As a people watcher, I was always interested in how the world works.  I was brought up on a farm in the country, so I didn’t have access to broad education, but when I went to college, I had a full right scholarship so I could study anything that I wanted.  Now, I actually have degrees in philosophy, physics and applied mathematics. While I was studying those things, I got very interested in complexity, which is basically the study of how unexpected things happen, and how they can arise from very simple inputs.
When I was starting to study complexity, I entered a collateral program in atmospheric sciences. I got exposed to global warming and what’s called anthropogenic climate change, which is changes in climate patterns caused by human activities. Somehow in the midst of it all, I realised that another physical scientist (I was working as a physicist at the time) studying climate change wasn’t really going to be the critical factor in helping us address our global ecological problems.
EWTT: What did you realise to be the critical factor?
Joe Brewer: I realised that the big problem was human behaviour that is prominent in culture, values and norms of society, the way our institutions are set up, our politics and economics – all the things that have to do with how people make decisions every day. When I left the university with a Masters degree in Atmospheric Sciences, and a strong interest in learning about human behaviour, I started taking in what is called cognitive science. It’s a cross-cutting set of approaches to understanding the human mind, human thought and behaviour. It includes psychology, brain research, linguistics, anthropology, computer science and several other fields.
Universities are mostly set up around disciplines, so you might be a professor or study a program in history or economics or political science, chemistry or biology – always within a field of knowledge. I wasn’t in academia. I was out in the world, trying to solve a very complex problem. My focus was on asking, ‘what do I need to solve this problem?’  This had me moving across many different disciplines to pull together insights, analytic techniques and tools toward explaining how human behaviour works, always with an eye towards sustainability.
EWTT:  What have we learnt about human beings in the last couple of decades about our mind and emotions, especially if we’re not the calculating, rational individuals that economic models assume?
Joe Brewer: We have learnt an incredible amount. Before the 1970s, every time a person made claims about the human mind, they were basing it more on philosophical assumptions than on observations and science. What happened from the 1970s onward is that we had got specific enough about different parts of the mind to be able to study them with rigour using scientific method, as a result of which a huge amount has been learnt about the mind and about human behaviour.
One of the basic things, if you’re familiar with the works of the philosopher in the late 1600s, Rene Descartes, is the problem he articulated that is now known as mind-body dualism. It’s a problem of saying that if our bodies are physical but our minds (related to the intellectual and mental) are not physical, how can the mind and body connect with each other?
One of the big realisations that has come out of many of the fields of research is that the minds that we have are not separate from our bodies, they actually emerge from the physical world, and are continuous within our planet. The general name for that is the philosophy of ‘embodiment’.  Embodiment means that our minds are part of our bodily experience. They arise from the kinds of bodies that we have and the kind of brains that we have, the kinds of physical and cultural environments that we have evolved in as well as develop in throughout our lifespans.
EWTT:  How do you link this understanding to something as large-scale as public policy, or what you call ‘cognitive policy?’
Joe Brewer: I call it ‘cognitive policy’ –  cognitive referring to the way people understand.  I was working a few years ago at a thinktank in Berkeley, California called the Rockridge Institute with a famous cognitive scientist called George Laykoff.  He and I were looking at the language that people use and the ways that people think about environmental policy and our goal was to improve the way in which climate legislation is developed.  We were working with members of Congress, members of environmental organisations and in the midst of our attempts to explain why the human mind is so important for policy, we found that the language people were using to talk about policy was too limited. We needed to distinguish between different parts of policy. The way that we ended up breaking it down was that we separated policy into the material component and what we call the cognitive component.
The material component is the nuts and bolts of how the policy works and the material consequences of the policy.  Let’s say we’re talking about healthcare policy. The nuts and bolts might be something like if a person makes a certain amount of money, he qualifies for a certain service, and there might be a policy mechanism that says what that is. The consequence may be that people who make less money are getting greater benefits because they can’t afford health care.
What we call the cognitive component of policy is the values, the moral perspective that motivated people to hear about the issue in the first place and the ways that they understand the situation, how they characterise what the issues and concerns are, and how that reflects a deeper set of assumptions about the problems and what the solutions should be.
The difference is that the cognitive component of policy has to do with how people think about the world, what their concerns are, what they are motivated toward, what they consider to be right and wrong and good or bad in any situation – so basically the deeper motivations for a particular policy context.
Let’s take another example of climate legislation. One of things that climate legislation needs to be is popular, by which the majority of citizens, at least in a democratic country, need to support it. They need to want the policy to stay in place. The reason for that is that dealing with climate change is a long term problem. So when the policy is put in place, it has to last not just for several years but several decades.  It needs to be popular enough that it can’t be dismantled by an uprising of citizens nodding in a different way that call for it to be repealed. For them to find it appealing it has to align with their values and concerns.
EWTT: What happens if the public perceives it as a short term sacrifice even though there may be a long term benefit? Let’s take for example a carbon tax, as in Australia. What if the public sees it as a burden?
Joe Brewer: An important discovery made by George Laykoff and Mark Johnson in the late 1970s, is when you look at the way human beings use language we don’t see the world in a literal way. We understand the world through metaphors.  So if you look at carbon tax, the important thing to think about is what metaphors are used to think about taxes.
I’ll give you two examples of metaphors that could shape whether a tax is popular or not. One would be that a tax is a burden, and you don’t want to be burdened. So a tax is something you want to minimise or get rid of.
Alternatively if you use the metaphor that taxes are an investment then it’s a different way of thinking of taxes. Then people might see that there are benefits that they get from society that only come about because everyone is investing in the infrastructure of society.  You may say you have an educated workforce if you invest in public education.  You may say you have public safety if you invest in medical science and hospitals.   You have a safe and fair society if citizens invest in courts, and law and contracts.
So thinking about what makes a policy popular is partly about which metaphor people understand the policies around.  Do they understand tax policy as a burden or do they understand it as an investment? In one sense it’s negative and in another sense it’s positive.   It’s an important way of thinking about the carbon tax in Australia , whether it’s popular or not will probably depend on whether the citizens of Australia are feeling that they are investing in their future to make their economy and environment more resilient and robust in a time of change and uncertainty, or do they see it as needing all the money they can get now. If they see the taxes as a burden to them now, then it is taking away their ability to get what they need.
You can imagine the significance now of media and of advocacy where there will be groups that will advocate for people to think about taxes as a burden , and another group for people to think about taxes as an investment.  An important thing to keep in mind is that taxes are simply not one or the other, but how people think about them shapes their appeal. The advocacy for one way of thinking versus the other will have significant impact on whether people support the policy or not.
EWTT:  Is it possible to make a tax or the removal of a subsidy, sound like an investment, especially in a developing country like India where there are huge inequalities of income and where there are millions of poor people can’t think beyond making both ends meet?
Joe Brewer:  There are two important observations that are helpful in addressing that issue. One observation is that people have a hierarchy of needs.  They are only going to be able to think of new wants and ideas and take time to build perspective if they have full bellies and safe environments. If someone is starving and is immersed in danger including the danger of sickness or the danger of violence, he or she is going to have a very hard time thinking about larger, more nuanced issues. So one challenge in addressing the tremendous social injustices that come about with sustainability is that many of the people, are also in a position that they are least capable of doing long term planning that is strategic, as their basic circumstances is about very hard survival.
Another challenge is that we are hardwired in a way that makes it easier for us to see simple relationships better than complex relationships, which are sometimes called systemic relationships. To be able to change fossil fuel systems and energy systems, as well as address tremendous inequities in society and deal with the political ramifications along the way, one of the big challenges is that we need to be able to see the nuances of the system and then redesign them so that they begin to work better.  That’s a difficult challenge – as human beings have a very hard time seeing systems.  We tend to simplify systems – in what is called a metonomy.
A metonomy is where a part of something stands for the whole thing. For example, if you’re at a restaurant where the waitress says, ‘the ham sandwich didn’t leave a tip’ it doesn’t literally mean a ham sandwich, but the person who ordered and ate it and left without leaving a tip. That’s just using the word, ‘ham sandwich’  to represents a person. Or when one says Delhi talks to Calcutta, it’s not the cities talking to each other but the leaders. In each case we use a simplification to represent something more complex. So when we’re dealing with these very complex problems, we have to basically counter our tendency to oversimplify. When we talk about poor people , who don’t have access to quality education, who don’t have the emotional resilience that comes with safe environments, they don’t know whether they’re going to sleep tomorrow, these are complicating questions if we are going to approach the kind of issues you raised.
EWTT: Politics and economics go hand in hand, and powerful interest groups and lobbies create resistance to change. What then is going to bring about change?
Joe Brewer: At first pass, changes in society are always lead by culture, so if you change the culture, you change the politics, you change the economics. It’s actually much harder to use politics to change culture.  One good example is how in 1865, all of the slaves were freed. It was a hundred years later that in 1965 that the Civil Rights Act was passed. There was a legal change, but the culture hadn’t changed yet. And then it took a hundred years to change the culture enough to accept a change in policy that would secure the rights, that would make it legally defensible to protect the rights of minority citizens. We only started dealing with racial inequalities in a systematic way since the 1960s onward, so the culture changed first then the politics followed.
So one thing that is very important for us to think about when we talk about changing behaviour is that ultimately yes, we have to change our politics, we have to change our economic systems because the way they are set up now cannot lead to sustainable outcomes, but to make those changes in political and economic systems we have to look at culture, we have to look at the stories that people tell themselves about where they come from, and what it means to lead a good life.
There has been a major global trend in the last century which has been the rise of global consumerism and consumer marketing.  Consumerism tells us stories of opulence and material success as measures of meaning and quality and happiness. So those stories are antithetical, they are the opposite of what we need to have to lead to a sustainable outcome. So we need to change consumer culture, that’s something that gets deep into the lives of people. When we talk on a global scale, it doesn’t mean we need one big monoculture that’s the same everywhere, but we can celebrate the unique features of different cultures that are resonant with sustainability.
India, I discovered during my visit, is an incredibly diverse country with so many different subcultures and languages and religions. Biodiversity is actually very healthy for sustainability because one thing we need is resilience. We need to be able to adapt to changing circumstances and if all of the cultures of the world are too similar to each other and if the way they all align is not healthy, then we are more at risk, and global civilisation could collapse completely.  When we talk about cultural change driving change in economics and politics, then we can find the strength of culture at different places and bring them together and drive innovation by plugging into places where cultures come into contact with each other.
We’re seeing that now in this global social movement, that firstly has been called the Arab Spring and then Occupy Wall Street, people from different cultures are describing it in a local way. Members of this movement in Spain are dealing with issues that have to do with Spanish culture, people in Greece are dealing with Greek culture, people in Lebanon are dealing with Lebanese culture, people in the US dealing with US culture. At a deep level, they are taking the paradigm of the global economy and they are suggesting a different way that people can come together to solve their problems, which means they’re suggesting an evolution of culture.
EWTT:  You enjoy studying deep history, and since you’ve mentioned Arab Spring and Occupy Wall Street, do you think change comes about in a disruptive way, or is it a gradual process that leads to a tipping point?  How do you think change is going to come about now when we most need it, in the face of the climate crisis?
Joe Brewer: Well, the way that big change always happens is there are periods when it doesn’t seem like things are changing very much, and then are short periods of time when things change very quickly. You could think of it as a ‘preparation’ stage and a ‘release’ stage.
Looking at globalisation as a process that goes back to the 1500s, and even the 1400s, there was global trade and then in the 1600s, we saw the global rise of corporations and in the 1600s and 1700s we had the creation of nation states. We had these big structural changes over several hundred years. And then we had the rise of market economies – they’ve only been around a few hundred years. There have been markets, there have been bazaars, the bazaar in Delhi has been around for much longer than a few hundred years, but the idea of a global market economy has been fairly new, it’s maybe only four hundred years old.
Given this deeper context, what we’re looking at now is a big change that has been coming for quite some time. I actually go deeper than that, and I like to think of it as three major periods of human cultural evolution. There was the period before agriculture, where we mostly lived in hunter-gatherer societies, and then we had the period where people learnt to domesticate plants and grow food, and that allowed human settlements to form and to grow. An interesting thing about human settlements is that if you have more food than you need, then you can grow your population. And if you can grow your population, you need more land, and that is the dynamics of empire and conquest. So empire emerged from agriculture.
About 10,000 years into the age of empire is coming to an end in one of two ways. Either it is going to end by collapse of human civilisation, that we basically wipe ourselves out and there will be an Easter Island kind of story. Or we change to a different paradigm that is not conquest. Now conquest and empire is now called ‘economic growth.’  It’s the same thing. We have an economic model that requires that the value of the currency for the economy must grow. And then if it doesn’t grow it becomes static and collapses just like when your heart – your heart has two dynamic modes – it’s either beating regularly or you’re dead, and there’s nothing in between.  Once your heart stops beating regularly it or it becomes rheumatic. You have a heart attack. Either it starts beating again or you’re dead. For a growth economy it’s the same thing, it keeps growing at an exponential rate, or it collapses.
So the changes that have to happen have to happen at a very deep level, at the level of a paradigm.  An interesting thing that can make us hopeful is that a paradigm level change happens very quicky. It’s like an earthquake. There’s a slow buildup and then an unpredictable release, and that change, that dynamic of slow build-up of pressure and release is how all physical systems change their state of matter. It’s just like when you start heating up water, the temperature continuously rises to a level  where it very quickly goes from a liquid to a gas and vaporises the water. That time of change happens over a very small change of temperature, in a short period of time.
So what we’re seeing now with these global social movements, is an acceleration of change that goes back at least 3 decades. In a global sense, we can see the rise of the environmental movement, which started about a hundred years ago and catapulted in the 1960s with Rachel Carson, and what’s called the modern environmental movement. We’ve seen the beginning of the collapse of the empire with post colonialism, from the independence of India, the rise of nation states, and social democracies. Going back 70 or 80 years, fairly quick and big changes have been happening. Now it’s much faster still.
Let’s take ‘Occupy Wall Street’ – it has been incredibly successful, in a short period of time. It has been only with us for a few months and it has already changed the way that people talk about the economy and social issues all around the world. Now maybe Occupy Wall Street won’t lead to the changes that we need, but the scale of impact would have been very difficult to predict. Imagine you were sitting and watching the world  in the beginning of August 2011, you probably wouldn’t have anticipated that something like Occupy Wall Street would have come into being and have such an effect in the last few months.
That is an indicator of how quickly change is coming and the fact that change is coming quickly tells us that we are in the middle of one of those phased transitions.  Change is happening very quickly because the entire system is reorienting itself. I think there’ll be a much bigger, deeper change in the next few years.
EWTT: Do you think that social media has played a role in this?
Brewer:  Absolutely, social media plays many roles. Even one step deeper than social media is the global digital communications system-  the internet, satellite communication systems, mobile phones – the whole system, and in all of that, the software that lets people in creative ways – facebook or twitter or email or whatever else you’re  thinking of in terms of technology.
What that digital communication system does is it democratises information. If you go back and look at what the printing press did to organised religion in the 1600s and 1700s, where prior to that the Catholic Church in Europe did everything in Latin. All the information was kept secret from people as they didn’t know Latin. They were only told what the leaders of the church wanted them to know. With the printing press it became possible for a lot of people to learn how to read, and share information. That automatically changed the way that organised religion worked.
A similar change is now happening with the digital communication system, social media and the internet. Information is now being democratised just as profoundly as the rise of the printing press. The fact that we can have instantaneous communication and that we can organise ourselves at effectively zero economic cost really helps –  it takes very little time, money and energy to send a tweet, or post a link on facebook, and people can organise themselves around what they are concerned or passionate about.  That ability changes the fundamentals of the economy, as now the economy is now driven by what is called pull marketing instead of push marketing. It means that people are able to seek out what they find desirable, rather than selecting amongst the choices that are presented to them.  It’s much easier to find like-minded people and we’re seeing that in social movements, that people are able to organise themselves very quickly, in real time, sometimes with hundreds of thousands of people organising themselves over small periods of time, like a few hours.  Social media is allowing that to happen, and the organised powers in the political and economic system are not that fast , they’re not able to keep up. The pace of change in that dynamic is faster than they can control, that is what is causing the breakdown of the systems under control  – it is allowing change to happen.
EWTT:  What are your suggestions to become more effective at a community level?
Joe Brewer: One of the frameworks that I was honoured to get to be a part of was “Identity Campaigning.”  The basic idea of identity campaigning is that people’s collective behaviour in society is shaped by their social identities. Social identities are in two forms. One form is the way that individuals see themselves as good or bad, and the other is at a community level, where there are shared identities, and where there are role models.
It is important to consider social identities, because they include emotions. Think about the social identity of what it means to be a good parent. It’s going to change from one culture to another, one community to another. But the social identity is understood collectively by the people in the community. So as you’re thinking about how to be an effective advocate, be mindful of the social identities that you select, that you want to highlight, and want to draw attention to, both in terms of the social identities that are positive , that people will resonate with, that you think people will want to be like, and also the identities that are negative, the ones that they don’t want to be, that they would be against. In order to be effective, you need to orient people around a different set of social identities than they had before.
With the “rational actor” theory in economics we are taught that being selfish is good,  because if you’re selfish you’re being productive, and as a trickle down effect, it brings wealth to other people.  Now we’ve figured out, that doesn’t actually work, but unfortunately that idea is still very common. When people aspire to serve themselves,  the social identities they are elevating is individualism, and suppressing identities that have to do with their communities.
One way that comes out is that they feel responsibility to themselves but they don’t feel responsibility to others. To get people to feel responsibility to others, we need to remind them that they have identities that they consider to be already a part of themselves, like being a good parent. A lot of people will recognise that as being a part of their identity, that has a social responsibility component, that is responsive to the needs of others around them.
I think in a deep way, it’s all about activating empathy and compassion in people. The more that they feel compassion and the responsibility to act in compassion towards others around them, the more they will work together to solve collective problems.  As you’re thinking about social identity, one way to answer yourself is, out of the identities that I am elevating in conversations, which ones are increasing compassion and responsibility toward others and which ones are decreasing the same? Just asking that question will orientate your thinking quite a lot, and help you become more effective.
EWTT:  People have good intentions but they seem to be too preoccupied. The general refrain is, “we’d love to do more for others and the world, but we’re too busy.”  How does one respond to that?
Joe Brewer: One thing is a lot of it has to do with design.  We’ve designed a global economic system that treats workers like gears in a machine. Machines tend to get faster with each passing generation. We’ve seen people find themselves working more and more, and find themselves engaged in activities more and more which keep them very busy. One thing to remember is that systems change on their own time scale, and one thing we’re seeing that a lot of people are busy working, doing all the things they’re doing in their lives, and they don’t have time for others.  It’s difficult to come out of those patterns.
They often have really good reasons to be busy. Someone might be working two jobs as he wants to send their child to college.  What we’re seeing with a lot of the global social movements that are successful right now like the Arab Spring, is that capable, educated people can’t find employment and there’s a telling observation that these are people who would be too busy to save the world, except that the economy is not serving them.
One thing we have to recognise is that change will happen when the systems are ready to change. Our economic systems at a very deep level have been poorly designed.  They perpetuate injustice. They make gaps between the wealthy and the poor larger with time.  They eventually come to a point when they become unstable and break down.
If we want to influence the behaviour of people, we need to engage people proactively when they are paying attention.  But they may not pay attention if they are overwhelmed with information and they’re already busy. Our strategies may not be able to operate at a system level scale to be able to change what their choices are, but when the system starts to stall and change, and the old dynamics are no longer stable and people are looking for different ways to be, then there will be tremendous opportunity to engage with them in meaningful conversations about deep change.

Joe Brewer can be contacted via info[at]cognitivepolicyworks.com.  Follow Brewer’s works through his website Cognitive Policy Works.