CS Professor here: just yesterday I did the discussion of a course projects' (Parallel Computing), and one of the three groups that I did yesterday have clearly gone the ChatGPT way. They couldn't even understand the choices the LLM made regarding the architecture, etc. The way to "catch" these students is similar to what we did in the past when students copied from other students which is "to give them rope to hang" - ask for clarifications until they follow unintended paths that lead nowhere.
To fellow professors, when you're suspicious my suggestion is to appeal to their honesty (like "let's be honest, how much of this code is yours, and how much is ChatGPT's?") and offer some empathy and understanding (like understanding they may had multiple deadlines in the same week, etc.). Nevertheless, don't miss the chance to give them the lesson on how is the correct way of doing things. The way to catch these students is to find the same signs of yesteryear copying from other students (which in essence is what copying from an LLM is, although the number has increased because they found us professors unprepared for the volume).
The other two groups also used LLM but in a high-level and architectural way. They were clearly responsible for the code (even if they didn't wrote it 100% manually) and could explain their reasoning and strategies used to solve the problems.
Me and my colleagues still have a lot of projects to review, and I asked them to keep the score of the number of projects like these, but so far, the score is 1 in 3 (33%).
So all groups outsourced their thinking to an LLM. Will they have learned anything they can apply to different types of projects in future?
In your opinion where is the line on where LLMs are useful/harmful to learning?
In my school projects I've found them super useful for working with libraries. In the past it felt like the theory I learned was pretty low impact. 80% of my time was spent learning the quirks of a library. Now it feels like I can take theory and iterate over a ton of different solutions without having to worry about learning whatever library the professor requires. Basically I'm I feel like it lets me spend more time learning the thing I want to learn rather than all the busywork around it. Would be curious to hear your thoughts. Thanks!
Not in the same level, see my other comments..
> The other two groups also used LLM but in a high-level and architectural way.
Sounds more like the score is 3/3 (100%)
Would you have accepted them cooy-pasting code from libraries together to build their project? If not, why is using LLM generated code different?
Using AI to inform architecture doesn’t seem so different from googling architecture in this case. Architectural patterns are mostly well understood and well documented these days and are something that you could piece together via Google search pre AI. The thing that AI brings to the table that wasn’t google able in the past is code generation. Previously you had to understand the architecture patterns to implement them yourself, but now the AI can just do it for you.
> Would you have accepted them cooy-pasting code from libraries together to build their project?
Yes, if they are "responsible" for the code delivered, where responsible means they understand the code, the architecture, the decisions made, etc.
In this case, the students had to invent multiple strategies to solve a specific problem. The "successful" groups did a mix of generated and hand-crafted code (don't know percentages), implemented different strategies and knew their plus and minuses, could change the code in a timely manner to accommodate some of my requests, etc. The "unsuccessful" group couldn't do any of that.
I'm not anti-AI (and really, what could I do if I were?) since I use it myself, I'm just anti-slop, especially from my students.
But in reality I've been slowly transitioning from group projects (for a subset of the grade) to "practical tests", where they must implement a significant subset of a larger project in a 2h class. Still experimenting though.
I'd say most of my CS classes (in the 90s) had us write out code by hand, on paper, during in-class exams.
It was fine.
> Yes, if they are "responsible" for the code delivered, where responsible means they understand the code, the architecture, the decisions made, etc.
This is a good principle to maintain, I think.
I'm not a professor, but I manage a team of about a dozen people. The maxim I have is: "You're responsible for anything that hits git."
Don't care if the LLM generated it, or the LLM told you if it's a good idea. If you commit it, you are endorsing it as a good idea - so you're the one I'm going to ask about it. I see the same principle at work in your pedagogy.
> I'm not anti-AI (and really, what could I do if I were?) since I use it myself, I'm just anti-slop, especially from my students.
This hits. Especially this part:
> and really, what could I do if I were?
My completely unsolicited opinion: you're doing a responsible thing by teaching these students how to use AI as a reference, and keeping them honest about not using it as a substitute for their own critical thinking.
> and really, what could I do if I were?
I’m not sure what you mean. Have a no-AI policy and fail those who ignore it. If students lie, discipline them appropriately for cheating.
How could I realistically enforce it? These are projects students do at home..
Same as other instances of cheating? A lot of students copy from each other and don't get caught. If anything, LLM-isms seem a lot more obvious, especially if the students are trying to avoid thinking or doing work.
Anyway, I'm not a CS professor, so I'm just venting.
He explains why in his comment. Read it again, carefully. Or ask an LLM for an "explainer"..
As a professor, what's you opinion regard the Socratic method with LLMs? Would that be preferable over simply "give me the answer" prompt?
Everything that provides students with a workflow to think and to try to find solutions to a problem is much better than giving the answer directly! Unfortunately there will always be students that prefer to take the shortcut..
How could we "force" the students to use an LLM that confronted their doubts with more questions? We could tell them to start each chat with a specific prompt (to use the socratic method, etc), but they could eventually jail-break it..
But nevertheless, I like your idea! This is something that a document highlighting methodologies for students on how to use LLMs effectively could/should contain..
I'm curious on your opinion, as a professor, how should universities (not individual professors) adapt to LLMs?
As an undergrad, I hope schools move to educating students to use LLMs in a more responsible way. You can't put the genie back in the bottle, and resisting progress is futile, might as well use the tool we now have to help students learn even faster and better (e.g., making feedback instant and not answers, helping digest or split up material, checking answers).
I know opinions about AI at (not only at) my faculty are very mixed, but I think the answer is going to be in the rational mean, just like how technorealism reacted to the internet[0].
In our last program board sitting, some teachers said that they think programming as a job will be completely irrelevant in two years, while other pushed for more adoption. And meanwhile I know of some students that are basically only passing because of LLMs, and it's bad, like "leaving claude output in markdown files and finished source code on the faculty server in /tmp because opencode did so" bad. And our first year classes completely prohibit even sharing tests or talking about the solutions, which in my opinion a) makes people extremely asocial and atomized b) doesn't prepare students for real life c) promotes dishonesty.
Still, I think our university's thinking is in a stalemate, not wanting pure AI output and useless students, while also wanting to move with the times, and I doubt it's the only one.
[0]: https://web.archive.org/web/20081009111415/https://artefaktu... (absolutely amazing read, recommend it)
I did a course of parallell computing 6 years ago. I still dont understand anything about it. Probably would if I had used an LLM, since I would have used it as a private tutor.
What is the policy and guidance you gave the students regarding LLM use?
This is all so new, and caught us completely unprepared that there's no official university-level policies. Most of us are still navigating the waters and seeing what works and what doesn't work anymore.
I have colleagues that are teaching for more than 30 years, few years away from retirement, who suddenly have been confronted with a new way of doing things. Those are the ones that are still insisting on doing practical projects, etc. I've only been doing this for 20 years, and I'm quite lazy (worked previously as software engineer), so I've moved to those practical tests. I guess that there should probably exist a class or workshop to teach these students how to use LLMs effectively, but as I said, this technology and its implications is quite new.
Personally, what I did was to give them the "lecture" in the line of that they do not understand what the machine has generated, that is not the way a true engineer does, try to do some parallel with things like an LLM designing a bridge and civil engineers building that bridge, and a fatal flaw collapsing the all thing, etc.
In other words, we do not have a formal system in place, it's all talking and convincing them. Obviously it's a big enough problem that should deserve more investment in solutions, but we are all overwhelmed by other tasks. Maybe LLM studios should be held responsible for all these "disruptions" and provide solutions to problems they created! :)