The biggest obstacle I see for AI in science is not scale. It is that today’s most powerful foundation models cannot represent the topology of a solution space — a set, a tree, a graph — and so they cannot effectively reason about or plan through consequences. Add a reward that is sparse and long-horizon, where the feedback loop outlasts the reinforcement-learning setup, and agents get worse, not better.
The root problem is representational. When you flatten everything into a token sequence, you impose a linear ordering on data that does not have one. Topology does not care about order. It cares about connectivity, holes, branches — the shape of a space. None of that is naturally expressible in one dimension. A model can learn to approximate it implicitly, or lean on sophisticated positional encodings, but those fixes are brittle and uninterpretable. If your representation cannot encode the topology, you are planning blind: interpolating, not reasoning.
State-space models do not escape this. I find them genuinely interesting, and Mamba’s input-dependent dynamics are a real step forward. But an SSM is still a 1D recurrence. A richer recurrent state is not the same as the right inductive bias for geometric structure; you are still asking a sequence model to learn topology from data, which is a hard inductive leap. Architecturally, I expect better answers to arise natively in the right geometric domain: graph networks, equivariant architectures.
But architecture is not the only way out. Tool use and memory already relieve some of the pressure, because they externalize the topology problem. Instead of compressing geometric structure into weights or a hidden state, the model queries something that understands the structure natively — a graph database, a symbolic reasoner, a search algorithm. The model becomes a coordinator rather than the geometric reasoner itself. That does not close the gap. It is still a very meaningful practical mitigation.