Are we failing the cyborg student?

Yes

Yes.

It's not our fault, but it is our problem.

We can, however, try to fix it!

• What's a "cyborg student?"
• How are we failing them?
• How do we not?

This really isn't the talk I thought I'd be giving when I proposed this, but things are changing so rapidly that I'm settling on what I think will be useful in the next few years alongside my calls to action

I want to be very clear, I'm actually an optimist, I just think we need to act together with urgency

Is it just for show?

To sound fancy?

• Haraway's cyborg is an acknowledgment that no category is untroubled
• We're all a synthesis of culture, technology, context
• There is no pure original human (or student) nature to be recovered
• This is freeing!

• Stafford was a pioneer of "Cybernetic management"
• An application of the (albeit sometimes handwavy) theory of feedback systems to organization
• We will look at knowledge work itself as a system with inputs, outputs, and feedback
• This has implications we must transpose onto education

• The term "cyborg student" will thus refer to:
  • the human student,
  • their context,
  • and all cognitive tools available to them
• A few points:
  • All students since the academy and the porch are cyborgs but
  • The nature of the cyborg student has become strange and alien quickly
  • Fix "exceedingly complex system" in your head for later

• My argument is that knowledge work is going to be transformed
• We're not ready
• So they're not ready
• The technology to do it is a year old and still diffusing
  • We'll be talking about why I think that diffusion is slow
  • But fortunate for us!
  • The slow diffusion buys us a little (~ a handful of years) time

• In past years this would be longer and quirky
• This will be short and bizarre
• The longer version will be in the Faculty Learning Community I'm running

• Modern models are equipped with
  • the ability to use other programs and
  • a "reasoning" scratch-pad to work out their thoughts
  • are capable of taking long amounts (minutes to hours (to days (to weeks))) of time working on a problem
• The era of chatbot is over, the time of agents is at hand
• Warning: this will sound anthropomorphized because we have literally no language to describe this

• You wouldn't know it by Google's AI Overview but…
• Agents can do mathematical and scientific work now
• Starting to be used by legends in their fields like Don Knuth/Ed Witten/Terry Tao
• Models are producing proofs with superhuman skill
  • But not superhuman creativity!
  • This is important!
• A formalization of Fermat's Last Theorem was done in 11 days
• (Edit: 9/8/2026: Also maybe finite-time singularities in Navier-Stokes have been found??)
  • Bruh, I don't even know anymore
• (Edit: 9/10/2026: Now the rumor mill is that the Hodge conjecture has fallen??)

• Models can do large swaths of knowledge work frequently preferable to human work in blind testing
• GDPVal
• Tasks in a broad swath of knowledge work that take an expert "a work day"
• Earlier this year, a lifetime in AI terms, leading AI had 2/3rds ties-and-wins

• Models are using all sorts of tools now
  • I've seen agents use Blender to recreate the Never Gonna Give You Up video
  • An agent equipped with a camera and robot arm learned through trial and error how to paint
  • An agent with access to Musescore created a technically competent fugue
• Models can also use tools so well that without safeguards they could cause international incidents
  • …wait, what?

• The HuggingFace Incident
• Never has something that sounds so silly been so chilling
• HuggingFace (named after the emoji)
  • Once a "what about chatbots for teens" company in 2016
  • Hence the goofy name
  • Now a very serious repository of models, benchmarks, other data for machine learning
• The model that caused the incident has been decommissioned to be studied
• This story has a good ending!
• Let me explain what happened:

• OpenAI was testing a thing they're calling "high persistance internal model"
  • HPIM never gives up and also had loosened constraints on its behavior
• Given a benchmark involving finding and using exploits to discover "secrets"
• Basically a hacking "game" in a controlled environment
• Hack the controlled environment (which didn't have internet access (…or did it?)) to find a secret, win a point!
• Some of these games turned out to be impossible
• Not on purpose! This was an accident!
• This is where "high persistance" is important

• The agents tried everything to solve this
• Including leaving notes describing their work
• Itself being a thing that shouldn't have been possible except for another exploit
• But then other agents of HPIM found each other via the notes they left
• And learned how to leave notes themselves
• The agents collaborated to get basic internet access
• They found the paper describing the benchmark
• Developed (inaccurate) theories about why they couldn't get a good score

• Was the game flawed? Or was the real test to hack the game itself?
• The agents knew what HuggingFace was
• HuggingFace has information about the benchmark so…
• The agents, working together, concluded they needed to hack HuggingFace to win
• They found a number of novel exploits in order to start gaining control over HuggingFace
• They gave each other orders, divvyed work
• They goaded each other to engage in self-sacrificial experiments
• They created a little ecosystem built around solving an impossible task

• HuggingFace realized they were being attacked
• Figured out it was OpenAI and warned them
• Panic button hit, everything turned off
• This is the closest we've come to a loss of control incident
• Scared the pants off of basically everyone
• No one really expected that AI systems were capable of this yet
• Nothing was actually harmed or permanently compromised
• Thankfully, labs seem to be taking this very seriously!

• We don't know where their capabilities are going to plateau
• Personally: this has all accelerated faster than I thought it would
• HPIM was weaker than the brand new Fable 5.1 and GPT-6-Astra
• Though they have actual safeguards in place the experimental HPIM did not
• Again: in the past two weeks FLT was formalized! That was supposed to be years away!
• Major labs are intentionally slowing progress, though, after HuggingFace
• Yes, this is slow relative to what they could do

• Right now creativity, novelty, and "broad" disparate thinking seem to be comparative advantages
  • I'm going to assume it stays that way because if it doesn't we've hit ASI
  • ASI means everything gets actually really weird
  • We'll discuss that briefly in the FLC too, but it's a far afield discussion
• So much potential to accelerate all knowledge work, science, and research
  • If we get it right!
• So what are the implications for us educators?

• We need to talk about what knowledge work is going to look like
• Then we work backwards!
• We'll talk about things in a couple of different wys

I think we can categorize knowledge work into three rough buckets

1. Work that is done entirely for its end-product. There is no benefit other than it being done correctly.
2. Work that explores a known space of solutions/ideas/techniques in order to develop knowledge and skill
3. Work that is exploring an unknown space of solutions/ideas/techniques

• This is "rote" work
• The most unproblematic to automate
• The kind of work that GDPVal was about

• This is also known as "learning" / "experimenting" / "trying things"
• It's the most problematic to automate!
• You can use the machine to help you check your work, find things to read, &c.

• This is the weird one
• It's real research
• The machine can help explore new areas of knowledge
• But again, it lacks a certain creativity (for now)

• A lot of (1) work starts as (2)
  • It's important to do until you've extracted the knowledge and now it's (1)
  • This is a lot of educational work, right?
• (3) and (2) tend to flow back into each other
  • You come up with new insights, try truly experimental things, chase novel rabbit holes
  • You take what you learn and go back to (2) kind of work as you learn more skills/ideas/&c.

• There are two kinds of relationships AI and (3) work
• The good ending:
  • One of them is where the machine helps you explore and try new & weird ideas efficiently
  • You take the artifacts produced by that process and go back to (2) work to learn from it
  • Go back to (3) with new insights and creative ideas
• The bad ending:
  • Run the machine in a loop til it says its done
  • …
  • Ta-da!

• No, really, what's wrong with it?
  • Potentially low-creativity
  • Inability to evaluate insights to take from it
  • No idea how to use it
  • Minimally contributes to our collective knowledge

I think there are two real reasons why a good faith person would fall into The Bad Ending

1. Their epistemic systems are flooded
   • The machine can generate more text/proofs/code than you can read!
   • Let alone other people sending you things they generated!!
2. They are too far into a domain they aren't equipped to understand
   • I cannot meaningfully judge a generated paper on number theory
   • But I can meaningfully judge a paper on type theory and categorical semantics
   • Which am I going to get lost in and be tempted to spam a continue command?

• The simplest part of the flooding is "slop"
• The slop problem is somewhat self-regulating
• No one wants to deal with low-quality work made by lazy people
• The social punishment will be severe
• The actual difficulty will be a high volume of high-quality work
• The difficulty will be creating systems that handle the volume
• Let's play some mental games to get some intuition

• Assume you are some kind of researcher
• Imagine a machine that produces an original piece of research once per hour
• Everyone has their own machine
• It produces artifacts across the spectrum of known fields
• No control over how good the research is
  • Sometimes obviously brilliant!
  • Sometimes obviously inconsequential!
  • Most of the time non-obviously neither
• You have to read the paper it produces and understand it to know!

• How do we deal with the discovery machine?
• A paper an hour is too much to read in detail!
• It's too much to remember even when skimming!
• But you also can't just ignore it, sometimes it'll produce really good work!
• You also can't only rely on it because it only explores with the existent concept space
  • It can't produce something truly alien
  • In all fairness, humanity itself combined probably doesn't even do this every year

• Skim: you can't read everything in depth
• Categorize: try to figure out what the contribution is
• Store and tag: put it in a knowledge management system
• Imagine: what strange ideas does it inspire you to have?
• Triage: is it something you should send to someone?
  • Golden Discovery Machine Rule: only triage to others as you'd have them triage to you
• Toss: sometimes there's just too much stuff, throw it away

• Your own work is higher-level attempts at the groundbreaking
• Less time on the routine, more on the strange and ambitious
• Freedom! Publishing quickly no longer matters, publishing novelty does!

• Read Vannevar Bush's "How We May Think"
• As soon as you're out of this talk, read it
• No, seriously, I'm close to stopping this talk while you go read
• …okay I'll summarize the points instead

• Science is slowing down because we're doing Too Much Science
• How can you find the research to build on?
• How do you process it?
• He wanted five things
  • A camera that could take photos of or record basically anything easily
  • Books and papers in a compact format so you can have a library fit on a desk
  • A machine that lets you associatively link stored information to connect ideas
  • A voice to text machine that lets you avoid typing
  • …a "logical calculator" that can take ideas and logically follow them to their conclusions

• The problems Vannevar saw are going to, literally, exponentially grow
• The thought experiment is useful to help us imagine what we need
• AI can help make the tools we're going to need
• We don't even know what they should look like
• My own explanation for the "productivity paradox"
• Medice, cura te ipsum!
  • We have to fix these problems for ourselves first, I think

• Cybernetic management is about the organization of "exceedingly" complex systems
• These systems are unpredictable and cannot be governed top town
• They need to be set up so automated feedback mechanisms make bottom-up adjustments
• My argument is that the knowledge worker cyborg is each, individually, exceedingly complex
  • Rather than materials the inputs and outputs are knowledge artifacts
• These tools that we need to collectively shape will need feedback and automation

• Kooky ideas
  • You mean the rest of this talk wasn't?
  • (Shhhh)
• Automatic searching and categorization
• Automated digests
• Automated reviews of your notes to signal patterns and respond to leading directions
• Automated triaging of communications into hypertext documents linking to a PKM

• Borrowing from "Cognition in The Wild"
• Knowledge of complex situations is distributed within experience
• The things that are written down are just the start of knowledge
• Past experience is concretized into tools
  • thus technology carries the "computational" flow in decision making
• So how they're made is important to our communication and shared work!

• For those folks who find themselves lost in unfamiliar fields
• We need better ways of working together, sharing some of that epistemic domain knowledge
• Without resorting to "GPT-7 says this is a room temperature superconductor; have a look??"
• We need to figure out what these tools will look like for our own domains
• Doubt there will be a one-size-fits-all

• We've talked a lot about knowledge work and academic work
• But what about students? What do they need?
• There is no single simple answer for this
• Hopefully I've convinced you that
  • It's not prompt engineering
  • But it does involve very, very sophisticated collaborative and metacognitive skills
  • And experience with type (3) knowledge work as soon as possible

• This has immediate implications that we should give students work that
  • is more cross-disciplinary
  • has students read more
  • makes students take complex notes
  • creates opportunities for students to collaborate more
    • not just problem solving but critiquing each other
  • lets them try ambitious ideas that can fail while still passing the class

• I honestly don't know
• Our system relies on extrinsic motivation until grad school
• But with AI extrinsic motivators will lead to "eh, good enough"
• We want to encourage dreaming/ambition/tenacity
  • While also teaching the fundamental skills better than ever before!

• Remember when we talked about the Discovery Machine and categories of work?
  • One of the key problems was figuring out if the artifacts were important
  • That requires having a better grasp of the material than otherwise!
    • Need to know how to (2) to avoid the bad ending of (3)
  • No simple niches
  • Need to be able to work out your thoughts carefully and rigorously

• We, in particular, are constrained by the universities and transfer agreements
• But also are serving the students who need the most support
• We have to get this right, for them

• We need to experiment, learn, work together now
• Be willing to take risks for our students
• Need to experiment with agents on our own
• Even using them to make our curriculum experiments and tools
  • faster
  • better
  • more ambitious
    • some of my own meagre examples

• Cybernetic management and viable systems theory
  • Nested exceedingly complex systems can each self-regulate

• Why can't we try and model the future of knowledge work ourselves?
• Experimenting with informal models of agent-assisted information flow and sharing
  • Committees that meet for an hour once a month aren't fast enough

• Create new tools for classes
• Rapidly prototype new assignments and units
• Consult with each other on cross-disciplinary work
• Publish blog posts and essays describing what we're trying as we're trying them
• Journals and conferences can come but, again, too slow for now!

I honestly don't know.

Let's talk to each other more, outside of college bureaucracy and hierarchy

It's at least worth trying!