I had quite an opportunity this summer. The National Assessment Governing Board (NAGB), caretakers of the National Assessment of Educational Progress (NAEP), also known as the Nation's Report Card, were looking for people who teach Computer Support Technician (CST) programs at their schools (secondary or post-secondary) who would like to come to St. Louis, Missouri for a week to participate in a study that determines the minimal level of mathematics education a student would require to enter such a program.
NAGB is a Congressionally appointed board consisting of various legislative and educational experts who convene to determine how well the US is doing in terms of academics such as math and reading (other topics are assessed as well). There's a company called WestEd that contracts a lot of the work the NAGB requires.
I had no idea what to expect. Though I've had a lot of post-secondary math training (Calc I, II, and III and some differential equations), I know I don't have much of an aptitude for math. I'm not a good math test-taker: even though I can ultimately reason my way to the right answer 75% of the time, I can't do it in the time limitations of most tests. I seldom get math questions right, and I cannot for the life of me formulate an algebraic expression for a given situation.
Further, to be really honest, math bores me to tears. I don't like reading equations or talking about math. It's just not an interesting subject to me.
But that doesn't mean I don't recognize its importance in the classroom, especially with computer science topics.
As a sidebar here: we need to differentiate between a Computer Support Technician--think "Geek Squad" and you have the right idea: people who repair computers and provide customer support for computer users--and a Computer Science major who will wind up writing code or working as a hardware engineer someplace like Texas Instruments or Intel. These are two different people. The former doesn't require a lot of math, the latter needs tremendous math education. (The reason why the latter needs so much might be good fodder for another blog post at another time.)
In St. Louis the first thing we were presented with were the stakeholders who were actually going to do the hard-core statistical analysis and research work required for the week. They billed themselves as "psychometricians" a term I'd not heard before, or if I had, it was only vaguely stored somewhere in my memory.
There weren't just CST teachers at the event, there were also math, reading and, surprisingly, Heating, Ventilation, and Air-conditioning (HVAC) teachers there as well. The week would be spent with some groups assessing minimal reading requirements for entering a CST or HVAC programs and some who would be involved with minimal math requirements. For each group--reading and math--there were two panels: Panel A and Panel B.
The three ladies heading up the effort--all Ph.D.s--were surprisingly organized and extremely clear in their instructions for us. Two of the women, Jannae and Susan, gave small presentations welcoming us to the week, but the primary heavy-lifting, in terms of speaking to us about requirements for the week, would come from a psychometrician named Luz. She was quite clear in laying out the weeks' work for us, and in showing us the path we would follow as the days elapsed. I was amazed with the organization these ladies displayed. Instructions and expectations were clear and they took lots of extra time to make sure everyone understood the basic concepts they were talking about, the outcomes they were after.
We were introduced to the 2009 NAEP mathematics exam. In fact, we had to take it, just like the seniors in various high schools took it. (NAEP is not required, and is administered by participating schools.) We saw exactly what the kids see.
Next we were introduced to about half of the test items (our counterpart--Panel A: we were Panel B--had the other half) and went over them in fine-tooth detail. There were math teachers, CST teachers and a facilitator in the room. Our goal was to understand the types of questions (multiple choice and constructive-response across algebra, geometry and trigonometry) and determine which questions fit into the minimal requirements for those entering CST programs--whether at the secondary or post-secondary level.
We spent a good deal of our time also writing what was called the Borderline Performance Description (BPD): a document that exactly stated the Knowledge, Skills and Abilities (KSAs) a student entering a CST program would need to know. A doctorate of mathematics in Georgia whose professor of mathematics job is to teach middle- and high-school teachers how to teach math walked us through the shaping of the BPD. We went through three different drafts of the document finalizing it toward the end of the week.
We frequently gathered together in a common place where Luz taught some very basic psychometric concepts. I was quite interested when we were in this setting, as we were shown the behind the scenes thinking statistical people are involved with when shaping assessment items and determining statistical correlation. She drew S-curves and talked about their statistical importance, talked about the varying degrees of difficulty of the NAEP math assessment items, spent some time going through exactly why we were going to take several passes through the data, and eventually set what was called a bookmark when we were confident of the minimal cutoff score needed on the NAEP assessment to enter a CST program.
In fact, we would set three bookmarks at various times during the week, based upon new information, and going back through the test items, discussing the relative importance of each to our programs. We spend a lot of time looking at the test items, ultimately obtaining a fairly intimate familiarity with them.
The biggest revelation I came away with was the seriousness and dedication of these researchers. This coming from a guy who thought the joke "Sixty-seven percent of statistics are simply made up on the fly" was funny. Not only did they have intimate knowledge of the test items, the reasons they were developed the way they were, and the rubrics by which they were scored, they also had a passion for determining as closely as possible the singular nut of truth denoting the cutoff score.
They looked at the data "fifteen ways from Sunday." We heard all kinds of different ways in which the data could be compared and intellectualized. I learned a great deal about the way in which psychometricians think.
There was a Thursday revelation in which we were shown where actual 2009 NAEP test-takers stood with regard to our minimal CST requirements. We were shocked! Only about 18% of NAEP test-takers would be able to enter our programs, given the cutoff we'd currently established: bookmarking the place where we felt the test items were no longer relevant for students in our program. I should note that the folks from NAEP made it very clear they were never going to publish these minimal requirements as an insistent baseline for students entering CST programs. These were merely recommended math levels we had established. None of this work was even guaranteed to find its way into being published, let alone recommended.
So we reassessed again, and made our final decisions, setting our final bookmark, establishing our final cutoff scores.
But we weren''t done. Next the psychometricians compared Panel A with Panel B scores. It was surprising how differently we'd assessed things. My group was a little lower, the other group a little higher. And of course, the psychometricians would take this data and correlate it so that a definitive cutoff could be determined. In other words, our cutoff score would not necessarily be the ultimate cutoff score.
I left with the impression that people who develop assessments and, more importantly, those who analyze and correlate that data, are really serious individuals who have a commitment to the truth the results dictate.
My only issue is a nagging sense that this data isn't important to anyone. Why? Well, first of all, because the NAEP isn't required, and it's taken by 12th graders who really don't need to take it, nor are the majority of them at all interested in it. Moreover, there was no guarantee our results would ever be known, and it was made loudly clear that there would never be a recommendation by the NAGB that kids wanting to go into a CST program be required to have this set of minimal math knowledge. Finally, and I think this is very important, none of the NAEP stakeholders had ever heard of CompTIA, the de-facto standard in certification testing in the computer industry. Lots of who teach CST programs lobbied hard for NAGB to develop a relationship with CompTIA, as this would be a harmonious relationship. But I was saddened when I heard the math professor say "We'll never talk to CompTIA. They can come to us, but we won't go to them."
To sum up: I'm very glad Federal education stakeholders are starting to think about careers and back away a little bit from the "every kid should to to college" mantra. And I think setting of minimal mathematics and reading requirements for various popular career paths is a wonderful thing. But education talks a good game when it comes to partnering with industry and then, when its crunch time--i.e. when experts are telling you that you need to talk to CompTIA and you say words like "never"--the commitment just isn't there.
So here's what I learned: I think there is actually some real value in the statistical process these people are going through. I totally got it: understood how thoroughly they were committed to the accuracy of the data and the value we were providing. I felt genuinely needed and useful.
But if there isn't a commitment to alignment with what industry is up to, then I don't see how we're going to actually churn out kids who have the chops they need to be successful in a technology program, or obtain an industry certification.
Listen: If you want to work as a CST in the industry, it's important you have a college degree, but it's even more important you have, at minimum, a CompTIA A+ certification, and desirable if you also have a Net+ cert. This is the real value-add for those wanting to work in this industry. The math isn't as important as the certs.
If we're going to progress educationally, it's vitally important we have our eye on industry, but even more important that we understand what industry expects of those who desire to enter it.
No comments:
Post a Comment