Hi Brian,

Thanks for your comments.

We know that the impact of the Poincaré program on students (via their teachers) is rather large and statistically significant but we don’t yet know which sub-areas of mathematics have benefited most.

The three graduate courses for teachers offered by the Poincaré Institute focus heavily on functions and on the algebraic underpinnings of middle school math. These themes are introduced as a means of uniting a wide range of standard topics and exploring them more deeply. In this broad sense we aim to improve teaching and learning across the middle school mathematics curriculum.

A close examination of our course notes, videos, software and activities for teachers will reveal concerns and aims of a more specific nature. For example, we spend much time considering the transition from counting number to measuring number and to real numbers. As you might imagine, this entails helping students gradually move “onto the real line” and “into the Cartesian plane”. Throughout this evolution, students will need to expand their sense of number from ordinal and cardinal number, to points, and later, transformations that entail processes and such as displacement, dilation, reflection, and shears. They will also be focusing on variables and relations, such as covariation, among variables. All the while, we need to constantly shift back and forth between operations on pure numbers and those involving physical quantities and magnitudes. Graphs become the representatons of functions. Slope comes to be reconceptualized as the rate of change rather than the inclination of a line.

Given that these aims touch on so many topics of middle school mathematics, I’m not sure precisely where to look for differential impact. In any case, where we are aiming is one thing, where we are effectively making a difference can only be determined with the help of additional data.

David (co-PI)

Brian,
Our program takes an algebraic approach to Mathematics. There is not much algebra in the MCAS (MA testing system) for the lower grades, so the program may be having an impact on how the teachers teach a number of topics.

We have only been analyzing test results for the grades that we cover. We hope this will have an effect on the higher grades, as the students move up. I would not expect much if any change in lower grades, as these are taught by different teachers, often in different schools
montserrat

Hi Carolina,
Some of the changes we made, based on teachers’ feedback and our evaluation of course materials offered to the first cohort of teachers are described in the following paper: Teixidor-i-Bigas, M., Carraher, D. W. & Schliemann, A. D. (2013). Integrating Disciplinary Perspectives: The Poincaré Institute for Mathematics Education. The Mathematics Enthusiast, 10(3).
Changes were quite substantial from cohort 1 to cohort 2, in terms of mathematical content, presentation of materials, and the pedagogical activities teachers participate in.
Further improvements were implemented in the materials for cohort 3. These will be offered to cohort 4 teachers in grades 6 or 7 to 9, with very minor changes. We will also design a set of adapted materials for teachers in grades 3 to 5 or 6, following a district’s overwhelming interest in expanding the program to elementary school.
Ana

Hi David L,

Thanks for your questions. Montserrat addressed the first and third ones.
I’ll respond here to your second question.

\
Let’s consider some rival hypotheses for explaining the results, that is, for explaining why the students in the Poincaré districts significantly outperformed students in the comparison districts.

\
Rival hypothesis #1. Perhaps, from the start, the students in the Poincaré districts were stronger in mathematics than students from the matching districts.
Assessment: There is no support for this. Past MCAS performance was one of the criteria for matching districts to the Poincaré districts. Before teachers took part in Poincaré there were no differences in students’ performance levels.

\
Rival hypothesis #2. The students in the Poincaré districts may have been better poised to improve their mathematics performance. For example, the matching districts may have had different demographics, income levels, district size, or % of students on free lunches as compared to the Poincaré districts.
Assessment: Again, this seems not to be the case, since these very criteria was used (in addition to past MCAS performance) for deciding whether a district constituted a good match for a given Poincaré district.

\
Rival hypothesis #3. Perhaps the sample sizes were too small and the observed differences were due to chance factors.
Assessment: This hypothesis can never be definitively ruled out, regardless of the significance levels obtained. However, it is worth mentioning that for each testing year there were roughly 5000 students in the Poincaré districts and 25000 students in the matching districts. (There were 5 matching districts for each Poincaré district.) A Wilcoxin test on the group differences yielded W=144, z=2.68, and two-tailed p = 0.0074. The Spearman Rank correlation between % of Poincaré teachers in a district and performance advantage (over matched districts) was 0.54, with p =0.003. (The corresponding Pearson correlation was 0.577 but we reported the smaller of the two values. Also, the Spearman test seemed more appropriate given the small number of district-grade ordered pairs.)

\
If you can think of any additional rival explanations, please share them with us.

\
At the moment our attention is focused on the challenging issue of determining what features of the Poincaré program may be responsible for the observed differences.

\
Best,
\
David C

How about overall teacher quality or parental support were better for those students in the Poincare program? I am wondering how you can issolate the program as the most likely variable responsbilble for the observed differences. We know how complex the learning environment of a classroom can be. So, variables such as teacher quality (years experience, education, preparation, resources), admin support, community involvement, or parent involvement—-any of which might be influential. I am sure you examined these possibilites.
Your analysis and communication of your analysis is quite wonderful.
Congratulations on such great work.
davod

At first blush it seems reasonable to raise the issue that the overall teacher quality and perhaps parental support were, from the start, better for the students of teachers in the Poincaré program. (We only interacted with the teachers, not their students. It gets confusing because the Poincaré in-service teachers are also students, i.e. grad students.)
\
But then one has to wonder, How is it that each of the 5 MA Poincaré districts moved ahead of the matched districts? Was this by chance? If the students in the Poincaré districts had more support, why were the districts not performing better than peer districts in the past?
\
There is evidence that Poincaré teacher quality improved during and after enrollment in the Institute. We have RTOP observations from our independent evaluator showing very promising (and significant) gains during and 6 mod and 1 year after. (According to RTOP observers, they were more responsive to student reasoning, used more representations for a given problem, for example.) The mean gain for Poincaré teachers was approximately eight/tenths of a standard deviation upward.
\
We’re trying to improve our measures of teachers’ understandings of mathematics. It has taken us several years to be able to clearly identify the topics and issues we have, from the start, implicitly expected teachers to make progress in.
\
In the past, we employed items for which there was, unfortunately, a ceiling effect; too many teachers scored at the highest levels on items. This was not necessarily due to having had extraordinarily gifted teachers. Rather, the items were too easy (quite a few of the items came from our earlier work in Early Algebra much of which covered grades 3-5). We’ll be applying the revised assessment in Fall, 2015. 18 months later we should have a better read of the evolution of Poincaré Teachers’ mathematical and pedagogical expertise.

David C

Thanks for your kind words.
\
You are correct about the perils of inferring causality from correlation. I would like to think that our choice of initial districts was not systematically biased, other than the fact that we favored needy districts. But random assignment is a tall order. It assumes that we can determine which districts receive treatment and which do not. (I suppose that is not entirely impossible; we could in principle find 10 willing districts and then randomly select 5 of them and follow up on all 10.)
\
But a special opportunity recently presented itself. We came across a district for which well over half (69) of its math teachers enrolled, from grades 3 to 9. We accepted them all (as we have done in the past). They can serve as their own controls. If we find that the teachers/their students make substantial progress over the 18 months, that will be compelling evidence, I believe, that it is related to the Poincaré program. Now, there will be additional factors at play. For example, we expect the culture of the school to undergo some changes.
\
In Educational research, even in a carefully controlled study, one always needs to go beyond the data to construct a model of causality. It’s easier to construct such a model if we have systematic and trustworthy data. But any model that aspires to generalization also entails theoretical considerations.

David L. in the thread of postings above asks:

“How about overall teacher quality or parental support were better for those students in the Poincare program? I am wondering how you can isolate the program as the most likely variable responsible for the observed differences. We know how complex the learning environment of a classroom can be. So, variables such as teacher quality (years experience, education, preparation, resources), admin support, community involvement, or parent involvement—-any of which might be influential. I am sure you examined these possibilities.”

We’ve been discussing the possible role of variables that might be termed methodological contaminants. I’d like to turn the discussion to some of the variables we think are more likely to be responsible for improvements in student performance. We would collectively refer to these as teachers’ deep understanding of mathematical content and of the ways students reason about specific concepts and representations.

Our teachers take three graduate level courses, taught across three semesters. The courses are the result of an intense collaboration between mathematicians, mathematics educators, and physicists, and aim at integrating mathematical and pedagogical content. Special attention is given to multiple representations and to how mathematical reasoning and concepts can emerge from actions and reasoning about worldly phenomena. Most importantly, it adopts a functions approach to mathematics and to algebra that enriches and integrates the existing mathematics content across the grades.

(There are several things that distinguish our approach to functions from other, unsuccessful programs from the past such as the New Math. For the moment, I would highlight the idea of introducing formal representations, e.g. algebraic notation, after students have made some progress in making sense of problem situations based on their existing knowledge, representations, and intuitions.)

Teachers dedicate around 10 hour per week to course activities, reading notes, solving problems, watching videos, working with software apps, interviewing students about their ways of solving problems, designing, implementing, and analyzing videotaped classroom activities, interacting online and engaging in weekly meetings with other teachers and instructors.

Our extensive matching criteria, especially those based on mathematical proficiency, income, and ethnicity, make it reasonable to assume that teachers’ years of experience, education and resources, administrative support, community involvement, and parent involvement would not differ much between the two groups. What did differ were the nature and degree of the teachers’ preparation: Poincaré district teachers had 18 months of extra preparation, of a very unique sort, that was not offered to teachers in the other districts. I am inclined to think that the Poincaré program is the most likely variable (or set of variables) responsible for the differences we found in the MCAS results.

Hi Elizabeth,
Thanks for visiting. This was a nice opportunity to describe our work in 3 minutes. And then, in the discussion, we can add more information.
Ana

Welcome, Elizabeth, to the fourth cohort of Poincaré teachers. I’m glad you liked the video. We are looking forward to working with nearly all teachers in your district.