TEFL book reviews reach a new low
No, most certainly not my fellow TEFL.net Book Review pages enthusiastic amateurs, who have written some great recent reviews on Creative Poetry Writing, New Insight into IELTS and Vocabulary . The guilty parties are people who really should know better and in fact think they do:
Step forward Brian Tomlinson and the rest of his multinational knowitall crew in the July 2008 edition of ELT Journal, who have somehow managed to give textbooks a score. And here they are:
Quick Smart English – 61.9%
face2face – 56.4%
Straightforward – 71.9%
Just Right – 68.5%
New English File – 63.3%
Total English – 61.5%
Innovations – 67.1%
Framework – 71.8%
Now, that is stretching credulity for me a bit I must say, especially giving a figure with a decimal point! At least they tested the books with a large cross section of classes and measured how much the students had improved using each course, though. Oh no, wait a minute, that was just a dream of mine. This bunch read through the books and gave them a mark (with a decimal point, remember) without having, as far as we can see from the review, taught with them at all. That’s right, they decided that Innovations was 10.7% better than face2face without actually teaching with either of them. At least they backed up their claims with references to other books, though. There are, for example, an awful lot of references to books by Brian Tomlinson, who apparently agrees with Brian Tomlinson.
How did this come about? It came about because they know what it takes to teach students well- so much so that they can judge a book without actually using it. That would save the publishers a fortune in trialling if it was so, but perhaps they should have a look at my previous post before cutting trialling out completely?
For the record, I have taught classes with both and know for a fact that in my classes face2face went down 15.43579% better than Innovations, although that is variable on humidity and maybe (as a comment in my previous post points out) just a couple of other factors…
January 27th, 2009 at 8:11 am
Yes, Alex, that was an interesting piece and it certainly gave me pause. To be fair, they did mention the limitations of the study:
“A different team of reviewers would almost certainly produce a different set of criteria and evaluation results. In this sense, we reiterate the point in Tomlinson et al. (2001) about the limitations of pre-use evaluation:
Only a thorough whilst-use evaluation and a rigorous longitudinal post-use evaluation could reveal reliable evidence about the value of the courses in affecting learner attitudes and behaviour and ultimately in contributing to the development of the communicative competence of the learners. (p. 82)”
January 27th, 2009 at 8:21 am
So, what was the point of the whole thing, and where did the decimal points come from??
January 27th, 2009 at 8:31 am
How on Earth could you judge a book down to a decimal point? This is ridiculous. I rarely read ELT Journal any more, the articles generally offer me little that I don’t already know (and I don’t know that much).
I guess this is only valid if you have complete faith in their groupm of reviewers, which I don’t.
January 27th, 2009 at 8:48 am
Seems you’ve been remaned…
http://eltworld.net/forums/viewtopic.php?t=1787
January 27th, 2009 at 8:48 am
Or even ‘renamed’ (insert embarassed smiley here).
January 27th, 2009 at 9:06 am
Actually, I could do with being remaned
January 27th, 2009 at 11:05 am
Re: TEFL Book Reviews
Here’s the team who wrote the study:
Our team consisted of Dr Hitomi Masuhara from Japan, Naeema Hann from Pakistan, Yong Yi from China, and Prof Brian Tomlinson from England. All the evaluators have considerable experience of using adult EFL courses in various contexts around the world as well as research expertise in materials evaluation. The team was coordinated by Dr Hitomi Masuhara.
Email: hitomi.masuhara@gmail.com
Email: N.Hann@leedsmet.ac.uk
Email: y.yi@leedsmet.ac.uk
Email: brianjohntomlinson@gmail.com
Feel free to invite them into our discussion!
Re: David V’s BBB thread on ELT World
That’s an interesting topic. You might also want to point out that TI actually states that those positions are JOBS (with a ‘free’ TEFL cert) and that, in general, the BBB won’t actually allow a complaint about a JOB. (BBB actually states this on their website). I’d love to delve into all of that , but I feel that my time would be better spent working on the 6 Way Series rather than dealing with Bruce and what I believe to be his twisted half-truths…
January 27th, 2009 at 4:07 pm
Further proof that the higher up in EFL you go, the bigger the inclination there is to be a twit…all hail the Tefl celebs…
January 27th, 2009 at 5:06 pm
Bizarre! Why didn’t they just give each book a star-rating, like many other book reviewers do? And I’d like to see their marking scheme, too! It all looks a bit too detailed to me – decimal points, indeed!
January 28th, 2009 at 6:20 pm
sigh…We publishers would love it if there were some universally-accepted scientific way of rating textbooks, but this study doesn’t seem to rise to that kind of standard. Alex is absolutely right. If we could apply a formula and just ask 4 experts for their opinions, that would save us a lot of time and money!
I suppose I should at least applaud the effort…but then Total English didn’t come out looking so well, so I’ll hold off.
January 28th, 2009 at 9:05 pm
The main appeal of writing about this for me is that incredulity is a strong emotion, and let’s face it the only strong emotion you usually get when reading through TEFL magazines is boredom. I think that is the main value of TEFL blogging, prompting the idea that reading about teaching should make you inspired, angry, fascinated or disgusted rather than just doing your duty.
Looking at it seriously for a minute though, there do seem to be those caveats that TEFLista copied here and that my eyes glazed over while reading. This method could also theoretically be used if they just picked up on one aspect of the books (comprehensible and stimulating input rather than PPP would be one suggestion, as that seems to be their thing) and gave a number just for that. As it stands, I still see this as a sign of the increasing trend for TEFL theorists to get all prescriptive with the flimsiest of scientific backings (although I have no idea if those 4 people are guilty of that more generally)
January 29th, 2009 at 11:52 am
Why not apply that formula used at the beginning of ‘Dead Poet’s Society’. I’d love to see a graphical representation of a TEFL book’s usefulness.
On one axis I’d have the number of times a book refers to graded readers published by the same company, on the other the number of times the book claims to have been written for students of a particular first language.
January 29th, 2009 at 12:18 pm
[...] as is Scott Thornbury. Hugh Dellar, despite his ‘Innovations’ being worth a mere 67.1% by ELT Journal is another who has the whole presentation thing down to a fine art, as is Michael [...]
April 6th, 2009 at 3:48 am
Excellent debunking of pretentious gibberish!
While numbers and decimals add precision in engineering and the physical sciences, they are utterly out of place in evaluating books and book reviews. Yet this very misguided effort reflects the profound insecurity of the educators who want to be considered “social scientists” and want to reduce the art of teaching to a uniform rubric of bullet points. Further, they mistakenly dismiss the power of teachers and defer to the myths of “textbook content is king.”
Education, however, remains more of an art than a science. Teaching should be tailored as much as possible on the unique desires and needs of the students, and students do have different personalities and experiences. Far too many school systems and textbooks, alas, rests on a different set of assumptions, values, and practices.