Forgot your password?
typodupeerror
AI Education Science

Computers Could Grade Essay Tests Better Than Profs 323

Posted by timothy
from the mckittrick's-behind-this dept.
An anonymous reader writes "Robot essay graders could be the answer to grade inflation. New software being tested turns over the task of grading to computers — this article has an interactive demo of the software. One professor says the computer is far fairer than human graders, who get tired and become inconsistent, or play favorites."
This discussion has been archived. No new comments can be posted.

Computers Could Grade Essay Tests Better Than Profs

Comments Filter:
  • by sandytaru (1158959) on Sunday August 07, 2011 @12:38PM (#37014868) Journal
    I once got an F on a paper from a TA who wrote in the margins "How dare you try to say what Shakespeare was thinking!" Um, that's what literary analysis IS, to some extent. You try to place someone's written works within the context of their culture and society at large and reconstruct their thought processes and views on the world. But that TA was an asshole and had it out for me, and many of us complained about him bitterly for years afterward. The only person who got an A in that entire section was one cute girl.

    As long as the robo-grader also includes a plagiarism check, I'd be okay with it. My husband is a professor and most of his failed papers are a result of TurnItIn.com catching outright plagiarism.
    • Re: (Score:3, Interesting)

      by hal2814 (725639)
      The purpose of literary analysis is to analyze literature, not the author writing it. It was an asshole way to put it, but the TA was correct. It doesn't matter what the author thought or even intended. The only thing that matters is what the author wrote and what we can analyze from that.
      • by damienl451 (841528) on Sunday August 07, 2011 @01:21PM (#37015236)

        This is why most people don't take literary analysis seriously. There is a real human being who took the pain to write a 400-page long book. Presumably, he wanted to convey *something*. But apparently, we have to act as if the book came down from heaven and we can't try to discover what the author wanted to say?

        The worst manifestation of this is when some literary theorists seem to argue that *even the author* cannot interpret what he wrote better than anyone else. He's just another reader!

        This sounds ridiculous to me. Even if the author writes an essay saying "this is what I meant when I wrote this", we're supposed to ignore that and simply focus on the words of the work because this is all that matters in literary criticism?

        • by digitig (1056110)

          This is why most people don't take literary analysis seriously. There is a real human being who took the pain to write a 400-page long book. Presumably, he wanted to convey *something*. But apparently, we have to act as if the book came down from heaven and we can't try to discover what the author wanted to say?

          No you don't, but you can't say what Shakespeare thought, you can only say what you think Shakespeare might have thought. That's valid interpretation, and if you can back it up with the work of other critics then you're heading towards a supported academic position. If you claim to know what Shakespeare thought then everybody knows you are bullshitting because nobody does.

          The worst manifestation of this is when some literary theorists seem to argue that *even the author* cannot interpret what he wrote better than anyone else. He's just another reader!

          This sounds ridiculous to me. Even if the author writes an essay saying "this is what I meant when I wrote this", we're supposed to ignore that and simply focus on the words of the work because this is all that matters in literary criticism?

          The author (possibly) knows what [s]he intended to communicate. You find what [s]he actually communicated in the words on the page. They'

          • by mooingyak (720677)

            The author (possibly) knows what [s]he intended to communicate. You find what [s]he actually communicated in the words on the page. They're not necessarily the same.

            What was actually communicated is entirely subjective and will usually vary from person to person. What was intended is singular.

            • by digitig (1056110)

              The author (possibly) knows what [s]he intended to communicate. You find what [s]he actually communicated in the words on the page. They're not necessarily the same.

              What was actually communicated is entirely subjective and will usually vary from person to person. What was intended is singular.

              What was intended might be singular but it rarely merits academic study and is inaccessible (even if the author is on record about it, how do we know that they were accurate and honest?)

              The whole point about the humanities is that they are inherently subjective (there's a bit of a clue in the name), so learning to deal with subjectivity is an important part of a humanities course. What one person thinks is of some limited interest. The similarities and differences between what different people report findin

              • I wrote a response to this, but I think Slashdot ate it.

                Has anyone done good empirical work on similarities and differences in perceptions of literature, according to cross-cultural, demographic, or other factors? The greatest weaknesses I have seen in the litcrit I've been exposed to have been the lack of empiricism, the lack of taste, and the lack of ability to write well (in fact, the propensity to write quite poorly, despite the use of jargon). But perhaps my exposure has not been broad enough.

          • It's hard to know, out of context, whether the grad student grading the paper was making an insightful criticism or a stupid one. Stupid grad student tricks abound, but stupid undergraduate tricks are even more common. It rather depends upon the work under analysis and what the student actually wrote.

            With Shakespeare,, most of what we know about what he thought is from literary analysis of his plays and sonnets, supplemented with inferences drawn from general historical knowledge of the period, and what we

            • by digitig (1056110)

              And that's even before you get into the question of whether Shakespeare was written by Shakespeare or by somebody else of the same name...

              When as a grad student I did a close reading of an extract from King Lear, I was interested to note that the second folio version had a lot more of the sort of stuff that structuralists look for than the first folio version did, which at least raises the question of how much of Shakespeare's reputation is because of good editors.

          • by pclminion (145572)

            No you don't, but you can't say what Shakespeare thought, you can only say what you think Shakespeare might have thought.

            Uh... Duh? When someone says "Shakespeare might have intended to convey such and such," it takes a very special sort of pedant to assume the former and not the latter. I'm glad I studied a field that isn't full of such masturbatory bullshit.

        • by calmofthestorm (1344385) on Sunday August 07, 2011 @01:45PM (#37015430)

          This is a bit of a strawman. In high school English, it was explained to us as: humans write literature, and sometimes they have something to say. This doesn't mean that they are the final word on what broader meaning their work has, but it does mean they have a deep insight into it. So no, don't ignore authors, but don't expect appeals to their authority to be viewed as anything but a fallacy in and of itself.

          Or put another way: Read Robert Frost's "Stopping By Woods On A Snowy Evening" (http://www.iment.com/maida/poetry/frost.htm#stopping), (it's short). The dominant interpretation of this poem is that it is an allegory of old age and death. Frost, however, insisted that this poem was about nothing more than taking a ride through a wood on a snowy evening. Who's right? It's not an either/or. In literary analysis there are right interpretation*s* and wrong interpretations, but it's not like there's just one right answer.

          Or at least that's what I remember from my last literary analysis class taken, which was in high school many years ago.

        • by Anubis IV (1279820) on Sunday August 07, 2011 @02:41PM (#37015836)

          I remember reading a quote from Tom Stoppard (playwright for Rosencrantz and Guildenstern are Dead, among others), in which he basically said that writing is like packing a bag that he later takes to the airport. The people at the airport opt to inspect his bag and they start finding all sorts of things that he didn't remember putting in the bag, but he can't deny that they are there.

          There is an argument to be made that we should take art as it was intended, but oftentimes the benefit of something is other than what was intended (case in point: movies that are so bad they're good), so there shouldn't be a reason why we deny that line of thinking as well. That said, there should be a limit. Some people, particularly the sort of liberal arts folks we all love to lampoon, try to insert their own things into the bag, rather than finding things that were legitimately there in the first place. But if they're simply discovering additional, yet unintended, depth to a classic piece of literature that can help us appreciate it better? Yeah, I see no problem with that. It may be unintended, but that doesn't mean it's not there. Even so, we shouldn't ascribe more meaning to it than it's due.

        • On a practical level the author probably can't even remember after years of writing, editing, and moving on to the next thing. And while hints from the author, his other work, his biography, etc. can illuminate, an undergrad class doesn't have time for more than textual analysis.
      • by epine (68316)

        The only thing that matters is what the author wrote and what we can analyze from that.

        Taking that position, it makes it hard to understand why so many academics heap scorn on Wikipedia, which operates under a similar proscription: you can only write what you can source, even if what you wish to say is so obvious that no reputable source bothers to spell it out, in direct terms.

        You can not state the insidiously obvious on Wikipedia. I once tried to add a footnote to a term from computer science that has lo

      • Ah yes, New Criticism, because Leavis and his idiotic ilk took a look at the downfall of logical positivism and said "if only our discipline could show such absurd hubris with even less justification". The second you make normative statements about criticism, is the second you look like a moron.

        Before you mount a defence of your absurd position I have two words for you I consider highly appropriate given the context, assassination, Macbeth.

    • by WindBourne (631190) on Sunday August 07, 2011 @12:49PM (#37014956) Journal
      Back at NIU, I had a lit class in which the female prof on the tenure track nuked all 3 guys in the class. I mean we scored D and F. ALL of the women scored much higher. I got fed up with this and one of my dorm mates gave a paper of his that had earned a A+ from the head of the lit program. It got a D-. After the semester was done, we took all of my papers including the purposely plagiarized one and went to the head. Showed it to him. Apparently, a major investigation was done, and she was released after that. My grades were adjusted up to a B after the head had re-graded all of the men's paper (I gave the head the paper that I had done and let him grade it).

      Sometimes, things do work the way that it should.
      • I've seen worse, though not in Lit.

        At Carolina, the Engineering program had a professor who, one semester, failed -everyone- ...except for the guy who was from the exact same part of India that he was.

        Everyone there tromped down to the Dean's office and showed him the facts. They all got regraded, and the prof was not retained.

        But worse than that was the Thermodynamics prof who graded entirely on the curve. As in, the Bell Curve.

        The first test was six definitions and one problem involving steam. All but

    • I had a teacher that was reviled by just about the entire campus such was her utter lack of competence. I had initially thought it would be possible to go through her exams by bullshitting, but then I stumbled on something that made me understand some of it.

      I had an appointment with her for reasons I do not recall and she was grading exams still. As I waited, she started discussing with another teacher, careless about me and many other students listening in. After the discussion ended, she decided to raise

      • by mooingyak (720677)

        Not really related, but I remember a chem test where there was a 4 part question, with each part supposedly graded independently from the others. Each part was worth 2 points.I got the first part wrong, and then you need to use your answer from part 1 in part 2, and so on. Having the first part wrong, naturally the numbers written for the others parts were wrong, even though the steps taken to acquire those numbers were completely correct. He docked points for each part that had the wrong number as the a

    • by nbauman (624611)

      You learned an important lesson or two. Your TA was an asshole. Just because he's a teacher doesn't mean he's right. Cute chicks win (you did read Anthony and Cleopatra, didn't you?).

      You wouldn't have learned that from a computer.

  • After school (Score:5, Interesting)

    by digsbo (1292334) on Sunday August 07, 2011 @12:39PM (#37014870)
    I had a prof in literature who only graded well if you made your critical essay about sexual imagery. At one point I gave up trying to "be me" and went whole hog, way overboard, almost parodying the over sexualized essay. And I scored an "A" for the the first time. Lesson learned? Sometimes it's OK to tell the boss what he wants to hear and do it his way, as long as it doesn't cost you anything, and nobody gets hurt. And, of course, life's not fair.
    • by Opportunist (166417) on Sunday August 07, 2011 @12:54PM (#37014998)

      The lesson you probably learned is that buttkissing gets you further in today's society than delivering good work.

      Who said school doesn't prepare you properly for your adult life?

      • Re:After school (Score:5, Interesting)

        by furball (2853) on Sunday August 07, 2011 @01:15PM (#37015172) Journal

        The lesson you probably learned is that buttkissing gets you further in today's society than delivering good work.

        At the same time, if a customer tells you what features he wants and you keep not building it, are you surprised when he's unhappy with what you delivered?

        Feedback is a valuable tool. What you do with said feedback is up to you.

      • by digsbo (1292334)
        Funny, though I think you were serious. I'm still pretty hard headed and though my work's not exceptional I usually cause my bosses to have some degree of heartburn because, though I will do it their way, I make a lot of noise if I don't like it.
    • Re:After school (Score:5, Interesting)

      by Phat_Tony (661117) on Sunday August 07, 2011 @01:45PM (#37015436)
      Similar experience here. I got very good grades on my college papers. Later on I had a sociology class and got a bad grade on my first paper. I knew it was a much better paper than that. I talked to the TA and she tactfully went over some things I could improve that sounded mostly like BS she was trying to make up because she didn't know what she could tell me. After a pause I said "it is my concern that I can not get a good grade in this class without agreeing with the professor's opinions," and she replied:
      "That would be my concern also."

      On the first paper we were allowed to re-write it and resubmit it, and rather than picking a new topic, I simply re-wrote the same paper from the opposite stance, parroting back the professor's (in my view entirely wrong) opinions. I even included some egregious BS about how I'd learned so much and realized how right he was. I worried it might be over-the-top with the sarcasm, but I couldn't help myself. Anyone without an ego problem would have seen through it, that a college student isn't likely to have a total change of heart and (in this instance) change from being basically a libertarian to being a socialist overnight because their professor was so brilliant that they showed them the error of their ways. I was a little scared he was going to notice and call me into his office for submitting a sarcastic paper.

      I got an A. The rest of the class was a disgusting piece of cake. There was no reason to bother with hard work, insightful points, and original analysis. It wasn't even necessary to read the material (although I generally did for my own benefit.) I just typed whatever opinions the professor espoused in class, with fidelity that was borderline plagiarism, and it was an easy A every time.
      • Heh, I have a story like this from high school. Basically, I'd been put in the "scientist/non humanities" bucket from the start. Every time I tried to write something, whether it was a short story, a poem, or a critique, it got a crap grade. Points for figuring out set pieces like use of metaphor and that stuff, but "something was missing" if you get me. Or don't get me, because it was all BS.

        So one day, I wrote a story about a guy kicking a ball around with his grandkid. Only this time, I smothered it with

      • by digsbo (1292334)
        At least in my case the stuff I had to parrot was something I saw as silly instead of unjust. I am certain I would not have been (might not be yet) mature enough to deal with it the way you did. It is good sometimes to get the insider view of a philosophy you disagree with to be better able to argue against it, though.
      • by mgblst (80109)

        I had the exact same problem with my Uni Maths classes. The Professor had some silly idea about what all the answers should be, and unless you parroted back the "correct" answers, he marked you as wrong. What a douche. Only a foll pretends that there are numbers beyond whole numbers.

      • The purpose of a test is to see if you know the material that is being presented to you. If the material you are being fed looks like bullshit that's still what is required to be put in the tests and assignments.
        I'd say the sarcasm was probably noticed but didn't cost any marks becuase it was used in a way that showed you were paying attention.
    • Poe's Law relates to how parody of extremism can seem similar to actual extremism.

  • Accuracy (Score:5, Interesting)

    by Wowsers (1151731) on Sunday August 07, 2011 @12:39PM (#37014880) Journal

    I can't say if a computer is better than a human at marking, but in my engineering subjects, when my name was on the test papers I did not get very good grades (actually at least grade lower than expected). But as soon as all the students were given anonymous numbers the grades went up. Conclusion, the staff could no longer decide to give better grades to their pet students. So in theory, there could be many students who get better grades because there is no more favouritism.

    • I had to mark some student essays this year and, in common with the rest of the coursework, even though it wasn't anonymous, I had no idea who most of the students were. Most of them had been in my lectures, but even most of the ones who had asked questions or come and talked to me after the class never actually told my their names, so they were effectively anonymous. I had no idea who the guy who got 99% was until he asked in the lecture why he had a mark deducted.

      As to whether a computer can be more

      • by nbauman (624611)

        As to whether a computer can be more fair, if it can then you shouldn't be setting an essay. A computer can tell if you've listed a set of bullet points correctly, but it can't judge your understanding of the subject. For example, one of the titles my students could pick was 'Give five design patterns for concurrent programming and suggest when each would be appropriate'. Students got a reasonable mark if they showed me that they understood the materials I'd covered in the lecture. They got a really good mark if they showed me something I hadn't covered in the lecture and demonstrated that they understood it. How would you program a computer to make that call?

        I agree. If you look at the example of computer grading in TFA, you'll see that it is in effect a set of bullet points.
        http://chronicle.com/article/Can-Software-Make-the-Grade-/128505/ [chronicle.com] It's like a short-answer test with the numbers and paragraph marks deleted and run into a paragraph.

        If you're going to use an essay-marking program like this, you might as well give short answer tests.

        It's nice to get the date of birth and death in an essay, but today teachers are de-emphasizing facts that you can look up on

        • Interestingly, I had this discussion at work the other day. IMO, knowledge IS bullet points. Sounds weird, I know, but thinking back to every essay exam I've ever done, there always seemed to be certain things that had to be mentioned. How you mentioned them seemed to matter (subjective grading), but in the end, people who understand a subject are always gonna mention a certain set of ideas.

          My colleague thought otherwise. Unfortunately, I don't have the bullet points of why.

      • I had to mark some student essays this year and, in common with the rest of the coursework, even though it wasn't anonymous, I had no idea who most of the students were. Most of them had been in my lectures, but even most of the ones who had asked questions or come and talked to me after the class never actually told my their names, so they were effectively anonymous. I had no idea who the guy who got 99% was until he asked in the lecture why he had a mark deducted.

        As to whether a computer can be more fair, if it can then you shouldn't be setting an essay. A computer can tell if you've listed a set of bullet points correctly, but it can't judge your understanding of the subject. For example, one of the titles my students could pick was 'Give five design patterns for concurrent programming and suggest when each would be appropriate'. Students got a reasonable mark if they showed me that they understood the materials I'd covered in the lecture. They got a really good mark if they showed me something I hadn't covered in the lecture and demonstrated that they understood it. How would you program a computer to make that call?

        I think subjectivity creeps in when it comes to the borderline cases. If a guy comes up with 5 patterns and manages to explain them like you would, you're inclined to think he gets it. But if a guy comes up with 5 sort-of-like-it explanations, containing some salient points, while omitting others, what do you give him? It can be hard to judge, especially as some facts are more important than others. Suppose there's one major fact and a couple of minor ones speaking in favor of a given DP, and the guy mentio

        • Of course. The module I taught was 50% essay and 50% programming. Of the 50% programming, about 5% was following a set of coding conventions and the rest was marked by a script. Some students got a few bonus marks for doing things beyond the requirements, and a few got some points for having code that was nearly right, just with some tiny bugs, but mostly the marking there was objective.

          The essays were entirely subjective. The point of them was for the students to prove to me that they understood the c

          • Hey, I'm quite interested in high performance. Does your course have a website/notes/syllabus/anything? I'm one of those guys who learned by doing on the job, after uni. Got an MEng though.

  • by hal2814 (725639) on Sunday August 07, 2011 @12:40PM (#37014886)
    My essay grades in college humanities courses were terrible until I started trying to figure out the political slant of my professor (or TA if the TA is the grader) and wrote papers supporting those views (and to be fair, those views weren't always left-leaning ones). I went from a C paper student to a low-A paper student in the blink of an eye.
    • My essay grades in college humanities courses were terrible until I started trying to figure out the political slant of my professor (or TA if the TA is the grader) and wrote papers supporting those views (and to be fair, those views weren't always left-leaning ones). I went from a C paper student to a low-A paper student in the blink of an eye.

      That sounds like an excellent humanities lesson in itself.

      • by Dunbal (464142) *
        Eh, what difference does a grade make in the humanities anyway. You're still not getting a job at the end of it all.
        • by hal2814 (725639)
          So what degree did you get that didn't require humanities courses?
          • by Dunbal (464142) *
            Medicine. I guess you could consider the psychology, disaster management and ethics courses to be part of the humanities. I also guess you could consider my whole field to be "humanitarian" with one caveat - we actually do the work instead of reading/writing about it.
          • In Europe, most degrees are different from the US model in that they are less broad. So a math degree will be just math courses, no need to know who Shakespeare was. A literature grad won't need to know how to add two numbers together.

            • by Dunbal (464142) *
              I also find that although it's not a hard and fast rule, a literature grad will never pick up a physics text or try to understand some biochemical conundrum, however a great deal of science majors (myself included) also excel at music and/or enjoy literature.
    • by nbauman (624611)

      You don't have an obligation to agree with your teacher's ideas, but you do have an obligation to understand them.

      Once you understand his ideas, you should be free to prove you understand him by repeating his ideas, then explain your reasons for disagreeing, and come to a conclusion different from his.

      It doesn't always happen, but most teachers allow or encourage students to disagree with them, as long as they follow the academic conventions of supporting their arguments with evidence. (And as long as they

  • by DavidR1991 (1047748) on Sunday August 07, 2011 @12:40PM (#37014888) Homepage

    Consistency is a fair point, but playing favorites? Isn't this what anonymous marking codes/IDs are for? (Or at least, that's what happens in the majority of universities in the UK)

    • by Tacvek (948259)

      In the US with the exception of tests/exams, it is almost unheard of to not have your name directly on an essay. A few Universities or programs may do that, but the vast majority do not.

      On standardized tests/exams the essays never have the authors name.

      On professor generated exams, the grader usually can see the name, but professors setting up exams such that graders do not know the student's name, while rare, is not unheard of. If such an exam contains an essay, then the essay would be graded without knowi

    • by hedwards (940851)

      It's a tough problem to solve. Ideally the instructor would be giving out rubrics when assigning the paper and sticking to it. At least that way folks have some meaningful idea as to what they need to do to get the grade they want. And can ask questions if they follow it and aren't given an appropriate grade.

      The times that I get freaked out are when I've got an assignment due that's worth half my grade and the teacher hasn't bothered to spend any time explaining what exactly one needs to do to get a good gr

  • Great idea (Score:5, Interesting)

    by WindBourne (631190) on Sunday August 07, 2011 @12:41PM (#37014894) Journal
    but it really needs to check for plagiarism. I saw a load of it up at Colorado State.

    In addition, it would ideally be able to handle lab books. I remember grading micro-bio 201 lab books back in the 80's, and I was getting tired after the first 30. The second 30 was a pain. The last 30, well, we finished the grading at a pizza joint over beer. I suspect that was how grade inflation happens.
    • by nbauman (624611)

      If it would improve your ability to appreciate the quality of my work, I'd be happy to include a joint in my lab book.

  • by ub3r n3u7r4l1st (1388939) on Sunday August 07, 2011 @12:43PM (#37014910)

    The GRE exam uses software to grade the essay portion for quite a while, along with a human grader. If these two scores different by a point or more, then it is forwarded to another human grader and the final score will be the average of the three entities.

    That cuts the cost of running the exam, considering the cost of incurring an extra human grader.

    It will soon pop up everywhere at university level, when the budget cuts are everywhere.

  • Hasn't that always been the case? I can recount dozens of personal examples in undergraduate/graduate (high school was too distant, sorry! But nor did I really take that seriously) where outside the multiple choice or true-and-false realm, there is always that element of human favoritism and non-neutral judgement involved. Certain people would get a lower/higher grade on a paper/research project that had really close ideology, thoughts or facts, that matched the next person (all cheating trolls stay in y

  • by Co0Ps (1539395) on Sunday August 07, 2011 @12:50PM (#37014964)

    What's up with the mass media headlines? Reading the summary actually makes me dumber. It talks about "computers" like they are sentient and grades the tests instead. Having professors first strictly defining the rules, entering them into software and having a computer evaluate those rules is still "professors grading the essays". It's self evident that the grading is better if it's more strictly defined.

    Wow, I can build a house faster with this hammer. Headline: Hammers Could Build Houses Faster Than Construction Workers (In Cyberspace)

    • by nbauman (624611)

      Having professors first strictly defining the rules, entering them into software and having a computer evaluate those rules is still "professors grading the essays". It's self evident that the grading is better if it's more strictly defined.

      True. If you read the example, you'll see that these "essays" are more like short-answer questions merged into a single paragraph. The program looks for a few pre-determined keywords. http://chronicle.com/article/Can-Software-Make-the-Grade-/128505/ [chronicle.com]

      It might be more fun working a few dates and keywords into a paragraph than filling out short-answer questions, but the information is the same.

  • Fairer vs. Better? (Score:5, Insightful)

    by fuzzyfuzzyfungus (1223518) on Sunday August 07, 2011 @12:52PM (#37014978) Journal
    Unless they've made some impressive advances in natural-language interpretation in the past few years that haven't trickled out into other products, I'm a bit puzzled as to how this scheme is supposed to work.

    Even the (comparatively much easier) tasks of spelling and grammar checking result in a fairly steady stream of mistakes from computer systems. I can't exactly summon much optimism for the likely outcome of such a system trying to distinguish between a paper with a well supported thesis and a paper that contains some declarative statements, a few quotations, and the word "therefore" at intervals.

    On the plus side, it should be pretty trivial to get the machines to do the same lousy job without the slightest consideration of the student's name/status/cuteness/willingness to flatter the professor; but what use is purely objective execution of lousy work?
    • by nbauman (624611)

      I'm a bit puzzled as to how this scheme is supposed to work.

      They gave an interactive example.
      http://chronicle.com/article/Can-Software-Make-the-Grade-/128505/ [chronicle.com]

    • I once worked with some people who had a patent for automatically detecting how good a foreign language speaker's English is. It basically recorded the subject speaking English, divided the recording into pauses and speech, and then measured the ratio of silence to speech. That was about it.

      It could be these guys are using a similarly simplistic metric. The article doesn't really give any reason to believe their system is good, only that one particular professor who likes to teach large classes (up to 100
  • more classes need to move away from the written test and to a more hands on / maybe even no test class.

    That fixes 2 things the people who just cram for the test and pass but I have little to no idea about the content or how to use it. As well as people who know what they are doing and are bad test takes / not that good at witting essays. Also cut's down on the people who pay for paper / essays witting services.

    • by digitig (1056110)

      more classes need to move away from the written test and to a more hands on / maybe even no test class.

      Good luck implementing that approach for journalism and creative writing courses! In some fields being able to write well is a vital skill. Even in sciences and engineering, the person who can communicate their ideas clearly and persuasively in writing will have a big edge over one who has similar technical skills but can't write well.

      • by hedwards (940851)

        You've got that backwards. Written tests are of absolutely no value in Journalism or art. You might do a quiz covering something like ethics, but in terms of the actual practices you're not going to test that in any sort of useful way using a written test.

  • by sgt scrub (869860) <saintium&yahoo,com> on Sunday August 07, 2011 @12:57PM (#37015032)

    As someone who never effected the curve or caught the affection of a teacher, I welcome our new digital grader overlords.

  • Let's be honest here, everyone had at least one teacher where it was obvious what you had to argue for or against to get a good grade. My German teacher was an ex-army officer. Take a wild guess what position you should take when the topic is the role of the armed forces in the history of the nation.

    It was easy to get a good grade. Why? Because you knew, no matter how harebrained or outlandish your arguments were, if it was what he wanted to hear, a good grade was your reward. Simply and plainly. Once you l

  • by paleo2002 (1079697) on Sunday August 07, 2011 @01:11PM (#37015134)
    I wasn't aware we now have access to AI this advanced. Spell check and (maybe) grammar check are reasonable, but how does a computer assess a student's understanding and mastery of a topic? How does the computer recognize originality, creativity, or intuitive leaps? Can the software recognize an effective argument, a convincing solution?

    I'm a geology and earth science professor. When I give writing assignments, I'm usually more interested in the content than the mechanics. I'll tolerate a few spelling and grammar mistakes if the content of the essay or paper demonstrates that the student understands concepts presented in class and, even better, is THINKING about the implications.

    For intro. writing classes, where grammar and structure are the point of the assignment, computerized grading is understandable; especially if your school has you teaching classes with more than 50 students (which is another issue entirely). But, in my experience at least, proficiency at writing is not always directly correlated with proficiency at class material.
    • Re:Wasn't Aware (Score:5, Informative)

      by tgv (254536) on Sunday August 07, 2011 @01:18PM (#37015206) Journal

      I'm pretty sure no program is capable of this (and I've got a PhD in natural language processing). They might be able to check for a couple of easily scored factors, such as number of words, and consistency between paragraphs, but I'm pretty sure that there is no program that could distinguish between an essay and the same essay messed up to base reasoning on false assumptions. I think someone left out a pretty important assumption: such programs might be able to score fairer (meaning: with less bias!), provided the students did their best.

      • by Animats (122034)

        Take a look at SAGrader [sagrader.com] and see what it is doing. It's not grading "essays". It grades answers to narrowly focused essay questions. It's looking for key phrases. The student''s correct strategy is thus to repeat, exactly, the language of the textbook.

        • by tgv (254536)

          That's even worse, and it's not even essays (as the OP states). So the teacher has to provide the program with the "correct" anwers? That can be a pretty long list. It's probably going to be good at rating mediocrity...

  • But the objectivity of the grades has nothing to do with the problem of grade inflation. Professors intent on inflating grades will simply reduce the weight of tests as part of the overall grade and count class participation, homework, etc. more /or/ add a flat number of points across the board to the results of the computer scored tests.

    Grade inflation, after all, isn't simple bias. We're not speaking of professors grading up people (or views) that they like and grading down people (or views) that they dis

  • I'm not arguing that this is a good or bad idea, but it won't do anything to change grade inflation. In my experience (as a TA for a number of different classes), college professors look at the point totals at the end of the semester and determine the letter grade cutoffs by hand so that they have the grade distribution they want. I'm not saying they're going through and making sure specific students get a particular grade, just that they want, say 50% A's 30% B's and 20% C's and they'll put the cutoffs whe

  • AIs to grade the papers I would assume would result in some folks developing AIs to create the papers...
  • by artor3 (1344997) on Sunday August 07, 2011 @01:24PM (#37015256)

    First, for those who didn't read TFA, computers play only a small role on a handful of essays. Most of the article is in reference to having a 3rd party grade anonymized tests, rather than leaving it to the professor or TA. During college, I had a job as one of those graders.

    We worked for five hours a day in the evening, though we could leave early and get the full pay if we finished all our papers. Most of the tests would be on general topics, but occasionally we'd get tests that required specific knowledge. In those cases, only qualified graders could review them, and we were given cheat sheets to make sure we didn't make factual mistakes. Essays were generally graded on a 1-5 scale (or a 0 if the essay was a blank page or similar). Each essay would be graded by two people, with a third breaking the tie in the event of a disagreement. However, we trained to be extremely consistent in the grading, so disagreements were rare and never more than a one point difference.

    A few times a day, we would get fake essays intended to test our grading skills. For example, an essay that was supposed to be a perfect example of a 4 would be given to you with all the rest. If you gave it a 4, you get +1 point. Give it a 3 or 5, you get zero points. Give it a 2 or less, and you lose a point. If you accumulate a lot of points, you get a bonus up to 50% of your pay. If your total score goes too negative, you get fired.

    It was a pretty good job, as crappy part-time "work your way through college" jobs go. The best part was whenever we got to grade essays by little kids. They were harder to score accurately -- it's hard to look past the abysmal handwriting and frequent misspellings. But they were frequently adorable and unintentionally hilarious.

  • Have written a few things, not including all the crap I spew here, I can tell you that any judgements beyond syntax, grammar, and semantics are purely emotional. That is because all human reasoning boils down to feelings. If that bothers you or intrigues you or bores you, then there you go.

    When machines are used to judge people's essays in subjective ways, either their state machines will be unemotional and miss much of the contextual meaning of writing, or they will be patterned after some designers' em
  • My partner is just starting an MA teaching program, and she's been ranting a lot about the utter uselessness of grades and standardized testing. Apparently, there are decades of research establishing that standardized tests fail to measure anything but performance on standardized tests, and grades measure little besides conformism, self-discipline, and a lack of creativity. (And self-discipline is not always a good thing [alfiekohn.org] -- why are you working so hard at doing things you don't really believe are worth doing

    • by ADRA (37398)

      So your ultimate solution is what now? I mean the sun causes cancer, but unfortunately for everyone, its essential to live. Why is this relevant? Well, if standardized testing and general schooling is so bad, what do you presume we replace it with? Rich people could get personal tutoring by experts in fields that they find meaningful. The rest of us should just learn the job our parents taught us because we get really really good at that one thing. Globalization would die because hell, if you can't generall

      • You're still working from the assumption that standardized testing is actually a meaningful test of whether someone has learned a subject. That's widely disputed. There's abundant evidence that tests and grades fail to predict anything other than future success on tests and future grades. They don't predict future professional success or future happiness.

        Educators have lots of suggestions for alternative models of education; I've often heard something like my suggestion for teaching writing, for instance, a

      • by gl4ss (559668)

        standard tests are good because teachers on average are just average teachers - and that's pretty shitty, so if they got inherited tests, you at least as a student get to see some questions you're supposed to know how to answer/solve no matter how stupid the teacher is.

        and every time you buy something built on the other side of the world, you're gauging if they managed to do it or not. if you're hiring them based on their chinese state school grades, you're a fucking idiot and your project is going to burn.

    • by Culture20 (968837)

      why are you working so hard at doing things you don't really believe are worth doing?

      Because if everyone works hard at pretending to be firemen, cowboys, and ballerinas throughout their childhood, we'll end up with a lot of disillusioned firemen, cowboys, and ballerinas. Very few people end up doing things they really believe are worth doing. Even in IT, we may believe in our immediate goals (fix computers/design systems/plan networks), but few of us probably are 100% gung-ho about the company's products or goals. Thus the constant tests to make sure people would be good at following dis

    • And, in my experience, an MA teaching program is just about utterly useless in producing anything but people who rant about grades and want to "stick it to the man." I spent a couple of semesters teaching classes in the College of Teaching building at my university. It was an eye-opener. Students were completing graduate classes with a poster as the final project. Just a poster. On something like Piaget's theories of development. With glitter and stuff. In my opinion, and I guess you should get off my lawn,
  • Based on what I know about the current state of AI, the essays that computers can grade fairly are NOT worth writing.

    Students and professors should try doing something useful... Something that pushes human intellectual boundaries and imagination, explores human emotions, discusses ethics and moral issues. These are not topics that today's artificial intelligence can handle.

  • http://www.alfiekohn.org/teaching/fdtd-g.htm [alfiekohn.org]
    "... The preceding three results should be enough to cause any conscientious educator to rethink the practice of giving students grades. But as they say on late-night TV commercials, Wait -- there's more. ..."

    Key points:
    1. Grades tend to reduce students' interest in the learning itself.
    2. Grades tend to reduce students' preference for challenging tasks.
    3. Grades tend to reduce the quality of students' thinking.
    4. Grades aren't valid, reliable, or objective.
    5.

    • by clong83 (1468431)
      I actually agree with a fair number of your points, and don't believe that grades tell the whole story of what someone got out of a class. I think it can be arbitrary, and at it's worst can instill hostilities amongst the students rather than a sense of camaraderie. Case in point: Once I had a solid A going into a final exam. The final was cumulative, and although I had studied well, I had a fever and could hardly focus on anything. I failed the final miserably. It was weighted as 50% of our grade, so
      • Education has at least three aspects, in decreasing order of importance:
        * Personal growth in a variety of ways (including spiritual);
        * Learning what you need to know to be a good citizen participating in political life (including voting);
        * Preparation for doing specific useful vocational work.

        Modern schooling has so degraded the notion of education that most people think it is mainly about the third item, and that is the example you drew from. There is also a fourth aspect in practice of schooling that has

    • by cellocgw (617879)

      Short summary: Alfie Kohn's a jackass. Picking on just one point (6) : just how does he intend to evaluate the student's progress, capability, and absorption of knowledge?
      Grades are like the Force (TM): there's a light side and a dark side, and you need to know which is which.
      FWIW, I found great pride and satisfaction in knowing I successfully solved all the (physics or math) problems on a test. I also felt pride and satis. when a Humanities prof. gave me a high grade on a paper AND included comments

    • I strongly suggest examining methods and reading some meta-studies, as well as considering the differences between the very young students addressed in Kohn's essay and college students. At a young age, research suggests that positive reinforcement is more effective in producing learning. In early adolescence negative reinforcement becomes more effective.
  • by Dr_Ish (639005) on Sunday August 07, 2011 @02:30PM (#37015756) Homepage
    As a professor, I can attest that the diagnosis of the problem here is too simplistic and the proposed 'solution' here is unnecessarily complicated. While it is the case that TAs and insecure professors will often inflate grades as they are scared of student appeals, the solution is to employ most experienced professors. There are also relatively simple methods that can be used to prevent grades becoming skewed. For instance, it is easy to grade anonymously. Just ensure that identifying details only go on the first page and turn the work over and grade from the back. One can also compare class mean and median scores (and SDs) with the scores from other sections of the same class. Such methods can ensure fair and consistent grading, without grade inflation. I always use such methods to great effect.
  • Aren't they a culprit too in grade inflation debacle ???

    I was a TA in a far east university in an Engineering department. Generally I consider my self a tough marker, as I expect students to arrive at answers with right logical reasoning. Having said that, I usually had a partial blind eye for students who has genuine drive towards studies -- post grad research types --, because their future shouldn't be eclipsed by a one bad grade. Also I highly control the grade distribution, such that only 5-10% of the c

  • I can see how in some cases the computer would do a better job than a professor. In particular, ones that could not care less about teaching. I'm in Physics, and in one grad course there was an essay on an exam that I got a zero on. When I looked at the solutions, it appeared that the essay on the key was actually my essay with a few slight modifications. Two sentences of the short paragraph were my words exactly. When I brought this to the professor (who was also my advisor), he (a) couldn't remember my name (b) wouldn't even look at the exam (c) wouldn't discuss the answer and deferred everything to his grader, who was another grad student. The grader had better things to do and just handed my exam back to me and said, "that's what you deserve." This same professor, it should be said, makes psychotic Wikipedia self-edits about how his work "reconciles quantum mechanics with the Christian faith", rarely talks to other groups about his research (once one of his students came to me to ask a question about a problem he'd been working on for months--within minutes I identified it as being identical to a well-known NP-hard problem), and frequently "dumps" RAs he doesn't like by simply ending all communication with them.

    My point is, the professors and TAs that grade unfairly don't do so because they can't. They do because they don't care. When I graded essays, I had a list of things I wanted to see in a correct answer and how many points they were worth, and a list of things that I would always take off points for. Every essay had a column of numbers next to it and a copy of my rubric so that any student could see exactly what they got points for and what they may have been penalized for. Out of classes of over a hundred students, I rarely received any complaints except for students who were on the border of failing and were desperate for one or two points. While sometimes grading essays felt like a simple application of a regular expression, searching for the gems of knowledge, equally as important was the logic that led to that conclusion. Correct answers obtained through incorrect application of concepts weren't worth any points at all, and it would be difficult for a program to match that with any regular expression.

    I guess experience with bad professors did teach me one thing--despite having no passion for teaching myself, I would always treat my students like people and do my best to ensure that they got the best education possible for their tuition.

1 + 1 = 3, for large values of 1.

Working...