Rendered at 18:02:09 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
derefr 21 hours ago [-]
It solves for the same thing a "train of thought" (internal vocalization) solves for in the human brain: there's a lot of stuff you "know" as episodic memories of words you've heard/read, but haven't yet internalized via practice into reflex reactions. It's a lot faster and cheaper to acquire a memory of some words, than it is to acquire an entire new skill, so you end up picking up far more memories-of-words than concrete skills from a given training corpus.
Reasoning (i.e. bubbling those remembered words up into your mental context, and then using a special "trigger effectively arbitrary synaptic activations dependent on the content of the attention-masked context and an ongoing state" bridging reflex on them, a.k.a. "system-2 thinking") is how you leverage memories-of-words into useful behavior. Rather than executing native skills embedded into your synaptic weights, you're running a single very fancy native skill that acts as an interpreter for word-memory data, slowly and strenuously turning words you're holding in mind into more words (synthesis), or into associations to other memories (word-memories or otherwise), or into actions/outputs, etc.
(An especially useful "trick" of this system is planning, i.e. chaining the interpreter through "mere words" that conjure more "mere words" to mind until you end up at words that form a declaration of what you're about to do. You can then rely on a special part of the bridging reflex, namely the "instruction-following" reflex, to react in a way where your actions are constrained by the plan you've generated.)
fedpost 20 hours ago [-]
I get that this is the goal, but having looked at reasoning traces they kinda read like baby babble. The model is just kinda sitting there restating the question and itself over and over. It'll sometimes kinda state/clarify assumptions but it feels nothing like actual train of thought
impossiblefork 18 hours ago [-]
That can make sense though.
I remember years ago now, trying to feed a text in twice etc., hoping that the representation in the KV cache of the second would end up better. I don't think it worked all that well.
But maybe that is most of the function of restating things babbling. To look at things multiple times.
cowlevel 19 hours ago [-]
ISTR some researchers did a test replacing the reasoning tokens with ".", and reasoning still improved the results because dummy tokens are extra passes through the same network, allowing for more computation.
derefr 13 hours ago [-]
I'm confused on how/why that would ever work for a regular feed-forward transformer model, given that the "more computation" that happens per output token in a transformer, just gets discarded after that token is emitted. The only thing the iteration-loop-step after the dot sees is a context that now has a dot in it.
joaogui1 2 hours ago [-]
Nope, it sees the KV cache of tokens coming before it for every layer
cowlevel 5 hours ago [-]
Is it? I thought the intermediate attention layers cross-fed.
theptip 14 hours ago [-]
Dots are better than nothing, yes, but not as good as arbitrary tokens.
devmor 17 hours ago [-]
If you can find this research I would love to see it. That’s extremely interesting.
arthurcolle 16 hours ago [-]
"Let's Think Dot by Dot: Hidden Computation in Transformer Language Models" by Jacob Pfau, William Merrill, and Samuel R. Bowman
If all you have is words and you want to make your ball role to the right place in the loss landscape, doesn’t make sense to babble them in a way similar to blowing on the ball to direct ot.
derefr 13 hours ago [-]
AFAICT those are various intermediate representations it uses for what you wrote / things it already stated in other ways; and when you see it restating things multiple times, it's due to a kind of swapping where it's struggling to juggle those representations.
If you think about how the algorithms the LLM has learned to do what it does must be represented across its layers, each such algorithm must be spread across a certain number of layers (since there's only so much logic that can encoded in a single GEMM op.) But this includes the algorithms the LLM uses to understand (recognize, parse) the context; and the algorithms the LLM uses to turn its semantic idea of how it should respond into tokens appropriate to the "word game" being played at the moment. Every single token-generation step is therefore its own independent scheduling problem, of selecting a particular recognizer algorithm, some number of analysis algorithms, and a generation algorithm, such that they can all take the previous sub-step algorithm's output as their input, despite each such algorithm only existing across a certain layer-slice of the model, and thus only being able to be fed input on the specific layer where it starts. (Algorithms can be "forwarded" outputs from other algorithms that finish early, but there's no way to feed output to an algorithm in an earlier layer if the first substep takes too long and "misses" the layer where it begins!)
One consequence of this is that complex recognition/generation, and complex analysis, trade off against one-another. The more a given token-generation step tries to "think", the less-sophisticated an understanding that step can have of the context, and the less-sophisticated a speech pattern that step can have for its output. And vice-versa.
In all parts of the LLM's output other than reasoning, a trained expectation of consistently high fluency in both recognition and production is expected/required. So analytical capacity is always sacrificed.
But reasoning has no such requirement. And so LLMs learn that during reasoning, they're free to flip between advancing with "NLP-focused" steps vs. "analysis-focused" steps.
And because the LLM's "verbal IQ" is temporarily decreased during analysis-focused steps, for those steps to make successful progress, they need the assistance of previous NLP-focused bridging steps to rephrase anything in the context that's too complex for the simpler NLP algorithms that get used alongside those complex analytical algorithms, into intermediate representations that those simple NLP algorithms can handle.
And then, for associational-memory reasons, you might see more bridging steps where the "simple language" the high-analysis-low-NLP algorithms output gets turned back into "fluent language". And on it goes, back and forth, during reasoning.
sweetjuly 12 hours ago [-]
If this is true, I wonder why LLM safety people put so much weight on the ability to analyze the reasoning traces? If restating the same content multiple times has some useful state change inside the model, it's not obvious from the trace what exactly that is, and so the leap to "less interpretable" model designs is not so severe. Maybe there's more nuance here :)
lossolo 14 hours ago [-]
Because it seems they aren't using words the way we humans do. Instead, they're using them the way you would use numbers or logical equations. So the whole trace is just a search through solution space using calculations.
bitwize 17 hours ago [-]
It's copying a summary of what it needs to know to produce a set of tokens to solve the problem at hand to the front of its context window, so that it doesn't slide off the back and the model doesn't lose track of why it walked into the kitchen in the first place once it gets there.
bwestergard 21 hours ago [-]
Can you explain why your view, which is the opposite of the OP's, is correct?
"RL training directly raises the success probability , by spending a lot of compute on searching for successful paths and then training the model to reproduce them. The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference."
raincole 20 hours ago [-]
Not mutually exclusive. LLMs are kind of like a self-fulfilling prophecy. Once some technique is discovered, it'd be part of the training process and becomes more efficient.
dist-epoch 20 hours ago [-]
I think it also addresses the issue that there is a finite amount of computation you can do in one pass.
So you use one pass to compute one relevant word, let's say you are working on a sound analysis task, and you output "FFT".
Now in the second pass, FFT is right there at the input, so you can push that idea forward more until you are out of layers again and then you use the output token to prime the next pass.
htrp 20 hours ago [-]
>I'm a researcher based in San Francisco. I like to imagine where the world is headed, and what it takes to get there.
>I did my PhD in computer science at Stanford, where I designed long-context architectures for sequence modeling and memory. Before that, I studied math & physics at Cornell. Most recently, I was a research scientist at Cartesia, working on in-context tokenization and novel architectures (HNets).
Not the MSL Alex Wang
burnrate 17 hours ago [-]
[flagged]
piker 21 hours ago [-]
> Consider the problem: “what’s 1 +2 +3 +⋯ +10”? One solution is to add the numbers one at a time: 1, 3, 6, 10, and so on up to 55. Another way is to pair them from the outside in: 1 +10, 2 +9, 3 +8, 4 +7 and 5 +6 each make 11, and five pairs of 11 make 55. Both of these paths land on the correct answer, 55.
Watching my son work through difficult math problems has forced me into this realization. In my head there is one "correct" path, and I often have to stop myself from interrupting his chain of thought. He often is just getting there a different way.
zmgsabst 21 hours ago [-]
We might even formalize that concept and study how many different ways we can prove a claim, what they have in common, if two paths are mostly the same or very different, etc.
Much of modern foundations is about categorizing reasoning structures by such paths!
Buttons840 10 hours ago [-]
I've wondered if there's some kind of "universal shape" that can combine efficiently to express all possible algorithms.
NAND gates I guess? Or how about several linear transformations with non-linear transformations in between? (To those who might not get the joke here, this is what neural networks are.)
Neutral networks are built of such basic "shapes", but I wonder if we'll ever notice that "oh, all the big LLM models have neurons that, when combined, form X shape, and these over here form Y shape, and so maybe we should be building neural networks from X and Y shapes implemented in hardware".
vova_hn2 21 hours ago [-]
I'm pretty sure that the current approach of baking in chain-of-thought reasoning into weights is not the end of the line. It just seems to accidental and ad-hoc.
P. S. The animated bird is truly magical!
HarHarVeryFunny 21 hours ago [-]
I just realized the bird turns it's head to follow the mouse pointer.
vova_hn2 20 hours ago [-]
Yes, and the animation is made using an interesting technique that I can't immediately figure out.
For example, what is this [0] or this [1].
The code is not obfuscated and seems well documented [2]. Maybe, I'll read it later or have an agent explain it to me.
[0] kind of looks like a 1D cellular automaton running inside a voronoi diagram. With a green blob overlaid on top.
JoshTriplett 20 hours ago [-]
Also, you can ruffle its feathers with the mouse.
origini 20 hours ago [-]
Yeah really unique, lovely and well made
Shacharp 15 hours ago [-]
Would be even better if an epistemic framework were available for the model to follow. We don’t expect our language centers to do executive function. An arbitrarily long chain of thought isn’t gonna make the model suddenly have grounding.
robrenaud 13 hours ago [-]
I prefer sloppy things that exist and work, rather than clean things that don't.
The model isn't grounded, but it can be wrapped in a system that is.
Shacharp 12 hours ago [-]
So long as the wrapper is an instance of itself there’s no reason it wouldn’t work.
15 hours ago [-]
gertlabs 16 hours ago [-]
This is a great article. In our RL work, we think of reasoning as a skill -- a set of tools and protocols to leverage knowledge a model already has to improve the probability of creating knowledge the model doesn't have.
Although specifically defining reasoning as a skill is hard, we've seen success with partners who agree that you don't need to enable reasoning on every single rollout -- you get diminishing returns past a certain point (the model often already has a complete toolkit for how to explore ideas and run calculations inside of its reasoning budget). Instead, it's more compute efficient to spend some percentage of rollouts training models with reasoning disabled, on problems that they are calibrated to have a good solve rate on without reasoning. This improves the cognitive core/latent reasoning directly which translates to fewer total tokens needed to solve problems, and harder problems that a model can solve. And it's more realistic, considering that humans don't need an inner monologue to solve many problems (even hard ones, e.g. a flow state).
vatsachak 20 hours ago [-]
Alex also had the transformers as memory blog. Beautiful. Somebody give him the bag
tristenharr 16 hours ago [-]
This is fascinating to me, I imagine this is in large part the future. This along with the weight space programming work that came out a few days ago are where I think we are headed.
I wonder if ideally we could skip ALL the "thinking/reasoning" steps if you trained a model to jump from "question space" to "answer space". Imagine a diffusion model or something that learns to map the start of the path to the destination. Very interesting
robrenaud 13 hours ago [-]
RLVR is a large part of the present.
I suspect weight space programming is not going to be the way models are practically optimized ever.
I think this is the core thrust of the joint-embedding predictive architecture that LeCunn has been pursuing.
Schlagbohrer 10 hours ago [-]
I use Qwen3.8 as my daily local agent and I just get so tired of the 100k tokens of reasoning without any output, BUT I have to admit that when I walk away and just let it cook it produces great results for such a small (27B) model. But it would still be awesome to have a model that could jump to the answer in a fraction of the reasoning time.
dogscatstrees 22 hours ago [-]
Wang's argument contradicts the view that intelligence necessarily improves through more reasoning and undermines any assumption that recursive self-improvement (RSI) will automatically converge toward greater intelligence.
jrflo 21 hours ago [-]
Content aside, that's a beautiful webiste
usernametaken29 19 hours ago [-]
> Train the model to generate better samples
This is precisely what humans do. We use small world statistics to drastically reduce the search space of problems. Of course there’s the whole issue of contextual search which LLMs address in no way whatsoever, so even improving sampling won’t help much, as inevitably one starts to generate too many hypotheses (reasoning traces) to get a good average.
22 hours ago [-]
14 hours ago [-]
avereveard 21 hours ago [-]
Normalizes user voice to goal so the action has more chances to success you can see this on model that dont hide traces sending very badly spelled or incomplete sentences
FLeXMurphy 22 hours ago [-]
What is the library, if any, used for the equation/formula rendering? Looks tastefully done.
freecodeio 22 hours ago [-]
> The mathematics on this page is being rendered by MathJax, which generates both the text spoken by screen readers, as well as the visual layout for sighted users.
FLeXMurphy 22 hours ago [-]
Lovely. Thank you.
domelian 22 hours ago [-]
also very curious, visualization is great!
0c3ca83 21 hours ago [-]
Misrenders on my browser. The bottom half of the characters in the equations are partly chopped off.
ketothekingdom 15 hours ago [-]
sorry about that, i'll try to fix it. which browser are you using?
Initially, I typed "intelligence is a path in token space" but then I decided that was too abstract. In case relevant, I was typing on a device with Muse installed.
BobbyTables2 13 hours ago [-]
Why is LLM reasoning in English? Seems a waste to “think” in full sentences with articles and other “unnecessary” words?
why waste time say lot word, when few word do trick.
16 hours ago [-]
NickNaraghi 22 hours ago [-]
> reasoning as pathfinding through token space
Sounds like it's hillclimbing all the way down!
Cool framing. I'm excited for the parallel construction of ___ as pathfinding through weight space.
drfloyd51 19 hours ago [-]
I don’t understand this part: if we assume a function that can validate an answer… what do we need AI to do? The existence of a function that can validate an answer means we have a proven way of finding the answer.
snarkconjecture 19 hours ago [-]
SAT solvers are a counterexample, as is most of formal mathematics
devmor 17 hours ago [-]
Imagine you are in a room with a lamp that has one hundred switches. Any wrong switch pressed will short the lamp and break it. You don’t know which switch turns on the lamp, but you know that when you see it light up, the problem is solved.
In this situation you have a validation function: “Is the lamp illuminated?”; While you don’t know how to find the answer to how to turn it on: “Which switch illuminates the lamp?”
cowlevel 19 hours ago [-]
I have here a function that can validate that a proof proves P=NP. Using this function please find a proof that P=NP.
jibalt 18 hours ago [-]
> The existence of a function that can validate an answer means we have a proven way of finding the answer.
That's a remarkable assertion.
moi2388 12 hours ago [-]
Well you could theoretically brute force it at that point. Practically not so much of course, but still.
akavi 21 hours ago [-]
My understanding of the math is weak (to say the least), but intuitively what I'm getting from this is:
1) Pretraining creates a bias _towards_ "correct answers" with each inference step
2) A longer walk is more likely to land in the correct region, given it's easier to verify you have the correct answer than to create it
3) RL selects the subset of walks that lands in the correct region and tilts the distribution in favor of such walks
4) By 2, that trains towards longer walks
...Is that correct?
EDIT: Wait, but this contradicts the post's claim that "The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference.". This requires that thinking longer is not at all causal for correct answers, which contradicts 2.
jebarker 21 hours ago [-]
Yes. One point worth making though is that pre-training trains the model to produce high probability completions - in practice, most of what it's trained on is a good completion as a human would judge it, i.e. it's not factually false data, so you get "creates a bias _towards_ "correct answers" with each inference step", but the model isn't really learning what correct or incorrect means in any way a human would consciously think about it.
trinari 21 hours ago [-]
does that lead to much inside the box thinking by models? Do they try to follow the clearest path?
jebarker 21 hours ago [-]
It depends on what you mean by “inside the box thinking”. In one sense, you are always mostly sampling high probability completions and they are going to be close to things that have been observed during training. Low probability completions will likely be low quality too. But, in a case where the context is forcing the model to generalize, I.e. interpolate in the latent space, the most probable completion may appear novel.
HarHarVeryFunny 20 hours ago [-]
During training you're trying to find paths to success, and reinforcing those.
There is going to be some minimum length path (assuming the model is generating some semblance of real reasoning), but more long ones given the possibilities for less optimal solves and backtracking after errors, so it stands to reason that if you think longer during training you'll find more paths to success.
At inference time you are hoping to follow one of the reasoning paths discovered during training. You need to generate enough tokens to follow the path (or one of the paths), but continuing to think past this point can only be detrimental - if you are lucky the model will realize the error and circle back, but if you are unlucky it will snatch defeat from the jaws of victory and wander off to who knows where.
There was a story recently linked here about one model "editing the weights" of another and thereby improving it's math performance, which was exactly about this - the need to stop generating at inference time at the right point.
IMO LLM's RL acquired reasoning, even if it gets the job done, sometimes spectacularly (math, coding successes), is really a very dumb cargo-cult process. At inference time the model is just trying to copy what it "saw" at training, without actually understanding it, which may even include duplicating errors and subsequent backtracking that had lead to success during training.
ryanschaefer 20 hours ago [-]
Lot of Alex Wangs out there
simianwords 11 hours ago [-]
I think of reasoning as a closure over knowledge the model already knows
bugreport671 18 hours ago [-]
FYI all equations are broken (tops are cut off)
ketothekingdom 15 hours ago [-]
hey sorry about that, which browser are you using? it seems to work on chrome desktop and chrome iphone
bugreport671 14 hours ago [-]
Sorry, are we back to “This site works best in IE 6.0”? At minimum one should and easily could check in safari, if already on an ios device, or a mac. Preferably FF as well, of course, though I understand it’s too much to ask.
To the point, I first saw this in experienced this in brave on iphone, and wrote the comment some time after, from memory. I just checked again and it turns out that the /bottom/ half of equations are cut off, in both brave and safari on iphone. Not sure if my memory is failing or whether it switched around.
Not sure whether you are actually the author and don’t want to go around throwing accusations, but - If you care and if this isn’t the case - i now consider sites where author states “works for me in chrome” as very likely to be slop. That said, due to issue above i didn’t spend much time on it to make a judgement based on content.
0c3ca83 21 hours ago [-]
Mostly, it seems to be solving the problem of people thinking about things themselves.
apishare_api 18 hours ago [-]
[flagged]
Nevermark 14 hours ago [-]
[dead]
Marha01 20 hours ago [-]
[dead]
Alice_0307 20 hours ago [-]
[dead]
niemandhier 21 hours ago [-]
That is how I have been thinking about LLM reasoning pretty much since it arrived.
I am not sure I support his statement about RL “searching for successful paths and then training the model to reproduce them”.
Don’t we train the model to use a certain method to explore the space in search for an answer?
As soon as you have tool calling and a way to verify your result, such as a theorem prover for example you are essentially running an optimization algorithm.
jebarker 21 hours ago [-]
> Don’t we train the model to use a certain method to explore the space in search for an answer?
Not really - you just sample random high-probability completions and then the verifier tells you which ones to nudge the model towards and which to nudge away from. There's lots of clever ways you could improve the search process, but standard practice right now is just random sampling.
Reasoning (i.e. bubbling those remembered words up into your mental context, and then using a special "trigger effectively arbitrary synaptic activations dependent on the content of the attention-masked context and an ongoing state" bridging reflex on them, a.k.a. "system-2 thinking") is how you leverage memories-of-words into useful behavior. Rather than executing native skills embedded into your synaptic weights, you're running a single very fancy native skill that acts as an interpreter for word-memory data, slowly and strenuously turning words you're holding in mind into more words (synthesis), or into associations to other memories (word-memories or otherwise), or into actions/outputs, etc.
(An especially useful "trick" of this system is planning, i.e. chaining the interpreter through "mere words" that conjure more "mere words" to mind until you end up at words that form a declaration of what you're about to do. You can then rely on a special part of the bridging reflex, namely the "instruction-following" reflex, to react in a way where your actions are constrained by the plan you've generated.)
I remember years ago now, trying to feed a text in twice etc., hoping that the representation in the KV cache of the second would end up better. I don't think it worked all that well.
But maybe that is most of the function of restating things babbling. To look at things multiple times.
https://arxiv.org/pdf/2404.15758
If you think about how the algorithms the LLM has learned to do what it does must be represented across its layers, each such algorithm must be spread across a certain number of layers (since there's only so much logic that can encoded in a single GEMM op.) But this includes the algorithms the LLM uses to understand (recognize, parse) the context; and the algorithms the LLM uses to turn its semantic idea of how it should respond into tokens appropriate to the "word game" being played at the moment. Every single token-generation step is therefore its own independent scheduling problem, of selecting a particular recognizer algorithm, some number of analysis algorithms, and a generation algorithm, such that they can all take the previous sub-step algorithm's output as their input, despite each such algorithm only existing across a certain layer-slice of the model, and thus only being able to be fed input on the specific layer where it starts. (Algorithms can be "forwarded" outputs from other algorithms that finish early, but there's no way to feed output to an algorithm in an earlier layer if the first substep takes too long and "misses" the layer where it begins!)
One consequence of this is that complex recognition/generation, and complex analysis, trade off against one-another. The more a given token-generation step tries to "think", the less-sophisticated an understanding that step can have of the context, and the less-sophisticated a speech pattern that step can have for its output. And vice-versa.
In all parts of the LLM's output other than reasoning, a trained expectation of consistently high fluency in both recognition and production is expected/required. So analytical capacity is always sacrificed.
But reasoning has no such requirement. And so LLMs learn that during reasoning, they're free to flip between advancing with "NLP-focused" steps vs. "analysis-focused" steps.
And because the LLM's "verbal IQ" is temporarily decreased during analysis-focused steps, for those steps to make successful progress, they need the assistance of previous NLP-focused bridging steps to rephrase anything in the context that's too complex for the simpler NLP algorithms that get used alongside those complex analytical algorithms, into intermediate representations that those simple NLP algorithms can handle.
And then, for associational-memory reasons, you might see more bridging steps where the "simple language" the high-analysis-low-NLP algorithms output gets turned back into "fluent language". And on it goes, back and forth, during reasoning.
"RL training directly raises the success probability , by spending a lot of compute on searching for successful paths and then training the model to reproduce them. The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference."
So you use one pass to compute one relevant word, let's say you are working on a sound analysis task, and you output "FFT".
Now in the second pass, FFT is right there at the input, so you can push that idea forward more until you are out of layers again and then you use the output token to prime the next pass.
>I did my PhD in computer science at Stanford, where I designed long-context architectures for sequence modeling and memory. Before that, I studied math & physics at Cornell. Most recently, I was a research scientist at Cartesia, working on in-context tokenization and novel architectures (HNets).
Not the MSL Alex Wang
Watching my son work through difficult math problems has forced me into this realization. In my head there is one "correct" path, and I often have to stop myself from interrupting his chain of thought. He often is just getting there a different way.
Much of modern foundations is about categorizing reasoning structures by such paths!
NAND gates I guess? Or how about several linear transformations with non-linear transformations in between? (To those who might not get the joke here, this is what neural networks are.)
Neutral networks are built of such basic "shapes", but I wonder if we'll ever notice that "oh, all the big LLM models have neurons that, when combined, form X shape, and these over here form Y shape, and so maybe we should be building neural networks from X and Y shapes implemented in hardware".
P. S. The animated bird is truly magical!
For example, what is this [0] or this [1].
The code is not obfuscated and seems well documented [2]. Maybe, I'll read it later or have an agent explain it to me.
[0] https://alexwang.ai/images/kea/head-fill.webp
[1] https://alexwang.ai/images/kea/body-feathers.png
[2] https://alexwang.ai/js/kea3d.js
The model isn't grounded, but it can be wrapped in a system that is.
Although specifically defining reasoning as a skill is hard, we've seen success with partners who agree that you don't need to enable reasoning on every single rollout -- you get diminishing returns past a certain point (the model often already has a complete toolkit for how to explore ideas and run calculations inside of its reasoning budget). Instead, it's more compute efficient to spend some percentage of rollouts training models with reasoning disabled, on problems that they are calibrated to have a good solve rate on without reasoning. This improves the cognitive core/latent reasoning directly which translates to fewer total tokens needed to solve problems, and harder problems that a model can solve. And it's more realistic, considering that humans don't need an inner monologue to solve many problems (even hard ones, e.g. a flow state).
I wonder if ideally we could skip ALL the "thinking/reasoning" steps if you trained a model to jump from "question space" to "answer space". Imagine a diffusion model or something that learns to map the start of the path to the destination. Very interesting
I suspect weight space programming is not going to be the way models are practically optimized ever.
https://x.com/NeelNanda5/status/2117375052618539485
This is precisely what humans do. We use small world statistics to drastically reduce the search space of problems. Of course there’s the whole issue of contextual search which LLMs address in no way whatsoever, so even improving sampling won’t help much, as inevitably one starts to generate too many hypotheses (reasoning traces) to get a good average.
Initially, I typed "intelligence is a path in token space" but then I decided that was too abstract. In case relevant, I was typing on a device with Muse installed.
why waste time say lot word, when few word do trick.
Sounds like it's hillclimbing all the way down!
Cool framing. I'm excited for the parallel construction of ___ as pathfinding through weight space.
In this situation you have a validation function: “Is the lamp illuminated?”; While you don’t know how to find the answer to how to turn it on: “Which switch illuminates the lamp?”
That's a remarkable assertion.
1) Pretraining creates a bias _towards_ "correct answers" with each inference step
2) A longer walk is more likely to land in the correct region, given it's easier to verify you have the correct answer than to create it
3) RL selects the subset of walks that lands in the correct region and tilts the distribution in favor of such walks
4) By 2, that trains towards longer walks
...Is that correct?
EDIT: Wait, but this contradicts the post's claim that "The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference.". This requires that thinking longer is not at all causal for correct answers, which contradicts 2.
There is going to be some minimum length path (assuming the model is generating some semblance of real reasoning), but more long ones given the possibilities for less optimal solves and backtracking after errors, so it stands to reason that if you think longer during training you'll find more paths to success.
At inference time you are hoping to follow one of the reasoning paths discovered during training. You need to generate enough tokens to follow the path (or one of the paths), but continuing to think past this point can only be detrimental - if you are lucky the model will realize the error and circle back, but if you are unlucky it will snatch defeat from the jaws of victory and wander off to who knows where.
There was a story recently linked here about one model "editing the weights" of another and thereby improving it's math performance, which was exactly about this - the need to stop generating at inference time at the right point.
IMO LLM's RL acquired reasoning, even if it gets the job done, sometimes spectacularly (math, coding successes), is really a very dumb cargo-cult process. At inference time the model is just trying to copy what it "saw" at training, without actually understanding it, which may even include duplicating errors and subsequent backtracking that had lead to success during training.
To the point, I first saw this in experienced this in brave on iphone, and wrote the comment some time after, from memory. I just checked again and it turns out that the /bottom/ half of equations are cut off, in both brave and safari on iphone. Not sure if my memory is failing or whether it switched around.
Not sure whether you are actually the author and don’t want to go around throwing accusations, but - If you care and if this isn’t the case - i now consider sites where author states “works for me in chrome” as very likely to be slop. That said, due to issue above i didn’t spend much time on it to make a judgement based on content.
I am not sure I support his statement about RL “searching for successful paths and then training the model to reproduce them”.
Don’t we train the model to use a certain method to explore the space in search for an answer?
As soon as you have tool calling and a way to verify your result, such as a theorem prover for example you are essentially running an optimization algorithm.
Not really - you just sample random high-probability completions and then the verifier tells you which ones to nudge the model towards and which to nudge away from. There's lots of clever ways you could improve the search process, but standard practice right now is just random sampling.