LLMs: Spot-On & Off-Target

I want to offer the framing I try to take on interactions with products like ChatGPT and Claude that help me make sense of moments of surprise, both positive and negative, or those instances where they are spot-on and off-target1.

The first is that these products do natural language search that you can not always count on (in contrast to key-word search). If they’ve seen enough content on a thing then they will give you more content on a thing. That’s it. If they have not, they’ll respond with anything that’s related, even if it makes no sense at all in relation to your request! They can not tell when they’re wrong. That’s an off-target moment: a response with content that’s peripheral, goes off on a tangent, or the opposite angle. Is it wrong or just retrieval gone somewhat wrong? How about a spot-on moment? Those “net new” discoveries people claim they have made that eluded a field for decades but turns out were hidden in plain sight2, for example, in these instances, it does something to the effect of concatenating fragements from several sources, to get from the starting point in your request to the finishing line in its response. Is that a discovery or just fancy retrieval?

The second is that these products are translators. We’ve had DSLs, compilers, and source to source translators of various quality for eons. We’ve had natural language translation for a decade at least. These products will translate from natural language to code, or from code to natural language, or indeed from one programming language to another. They can do this more arbitrarily than anything we’ve seen before (at least to my knowledge). “Product” people often point to the idea that they use these products to generate or translate (the code for) an artifact from a big long (perhaps incomplete and buggy) specification. Is that software development, product development, or just translation? I wrote the opening sentence in my first post hypothetically, but it seems these products really have become the primary way some people in some professions interact with computers.

Natural language is just one approach at any problem3. Others include formulae, diagrams, tables, code, etc. Have these products (finally) made natural language (a bit) more tractable for computers or just made the content on the WWW more easily but not reliably accessible4?

This post isn’t informed by data. It simply offers the framing and questions I use to help me make some sense of my own experience, others’s experience, and the chasm between the two. I hope they help anyone equally baffled on occasion.