FedSpeak

Three official publications, scraped from the Board of Governors' own archive pages: every FOMC statement and set of minutes since 1994, and every Beige Book since the HTML archive begins in late 1996. Statements are sparse early on because the FOMC only issued one when it changed policy; the spike years are 2001, 2008 and 2020, when it met between meetings.

Thirty years of the Fed's website is five URL schemes, four page templates and two character encodings. The scraper follows link labels, which have been stable, rather than URL patterns, which have not, and it hands raw bytes to the parser so 1990s pages in ISO-8859-1 decode correctly. Cleaning removes navigation, press-release framing, footnote markers and comment blocks, and maps typographic quotes and dashes to ASCII.

No byte-pair encoding. Every distinct character in the corpus gets an integer, which makes 86 tokens: newline, space, punctuation, digits, two cases of letters. The model's entire input is a stream of these integers, and its entire output is a probability over the same 86 for what comes next. Type anything and see what the model sees.

The model is Karpathy's nanoGPT, unmodified: 6 layers, 6 heads, 384 wide, 256 characters of context, 10.65M parameters. Every step takes 64 random windows of 256 characters and scores the model on predicting the next character at every position. Cross-entropy, backprop, AdamW, a cosine learning-rate schedule. That is the whole method. Loss is in nats per character; 4.45 is what uniform guessing over 86 characters scores.

The same run again with a checkpoint kept every 200 steps, each given the prompt The Committee decided to. Drag through the run. The order in which things arrive is the order of their statistical strength: character frequencies, then spelling, then words, then collocations, then sentence templates, then document structure.

Temperature doesn't change what the model knows; it changes how much of the probability distribution you let it sample from. Low shows what it is most sure of, which for this corpus is boilerplate. High shows the long tail, where fabrication surfaces as misspelling rather than as confident nonsense.

Every output here was generated once and saved, with a fixed seed, so the page answers instantly and shows what each model actually produced rather than a cherry-picked run. Five models: the character model trained from scratch, then SmolLM2-360M and Qwen3-1.7B, each untouched and again after fine-tuning on this corpus. The last prompt is a real question, included to show what a base language model does with one.

Eleven runs of 1000 iterations each, at a fixed 16,384 characters per step, on a 3.2M-parameter baseline so each one takes minutes rather than an hour. Every run changes exactly one thing. Characters per step are held constant when the chunk size changes, so the 16-character run sees 1024 sequences per gradient and the 1024-character run sees 16, which turns out to be the source of the most interesting result.

Re-running every one of them at the same seed moves the answer by at most 0.005, because GPU kernels aren't bit-for-bit deterministic. Differences smaller than about 0.01 on this page are noise; the ones the conclusions rest on are several times larger, and the closest call was settled with three seeds.

The first version of this sweep tested only 16, 64, 256 and 1024, and concluded that 64 and 256 were tied. Filling in the gaps showed a U with a minimum at 128. Then scoring every model fairly moved the minimum again, to 64, and showed that most of the left side of the U came from the evaluation, not the models. Given its full 16 characters, about three words, the smallest model predicts nearly as well as the best: most of what makes the next character predictable sits inside the current word.

Chunk size. Short chunks learn faster and long chunks learn further: the 16-character model leads until iteration 250, the 64-character model until 800, and 256 only draws level in the last 200 iterations. At 1024 the gradient is averaged over 16 sequences instead of 1024, and 1000 steps is not enough to recover from that noise. Each model is also scored in windows of its own chunk size, so this axis mixes how much context helps learning with how much it helps prediction.

Size. Eight times the parameters between 0.4M and 3.2M is worth half a nat, roughly the difference between "accommodity" and "accommodative". Another 3.4x to 10.65M is worth 0.12 more.

Learning rate. On the 3.2M proxy, 3x higher than nanoGPT's default wins by 0.045. Applied to the 10.65M headline model it lost: 0.708 against 0.698, and it trailed for the whole run. The stable learning rate usually falls as a network widens, which is the problem maximal-update parameterisation exists to solve. Without it, a rate tuned on a small proxy doesn't transfer, and the headline model keeps 1e-3.

The prediction going in was that statements would be hardest: they're 1.6% of the corpus. They are the easiest by a wide margin, for every model, because consecutive statements reuse whole sentences verbatim. The Beige Book, two thirds of the training data, is the hardest, because twelve districts describe different businesses in different words every six weeks. Loss per character measures novelty, not importance.

The loop is unchanged: windows of tokens, next-token cross-entropy, AdamW, cosine schedule. What differs is that SmolLM2-360M starts from four trillion tokens of prior training instead of noise, reads the corpus as BPE tokens instead of characters, and moves with a learning rate a hundred times smaller. Each document got a one-line header like [FOMC statement | 2010-01-27] so the result can be asked for a source and date.

On a 16 GB laptop the run was paced by memory, not compute: bfloat16 autocast, gradient checkpointing and a micro-batch of one were all necessary, and 320 steps over 1.3M tokens took 88 minutes.

A second run repeats the experiment with a model almost five times larger, Qwen3-1.7B. Full fine-tuning no longer fits, so that run freezes every pretrained weight and trains rank-16 adapters beside them, 1% of the parameters, saved as a 77 MB file. Qwen3-4B was tested and rejected: it pushes this machine into swap and would take about a day per run. The two runs are otherwise identical, same corpus, same headers, same token budget, and validation pinned to the same documents so a different tokenizer cannot shift the split.

The base model knows what the FOMC is and writes fluent English in the register of an internet comment thread. The fine-tuned model writes a statement: policy sentence, rationale, forward guidance, voting paragraph. It also reports a rate that rose to its lowest level, fills the roster with four real governors and four invented ones, and in the minutes sample quotes Mr. Trump at an FOMC meeting. Fluency went up by a lot. Truthfulness did not move, because nothing in the training signal refers to the world.

The larger model makes the same point harder to miss. Every Beige Book in the training data lists which city belongs to which Federal Reserve district, the same twelve lines, 112 times over, never varying. Asked for a Fed document, the fine-tuned Qwen3 wrote out that list with only the first line correct, assigning Chicago to the Second District and inventing an Eleventh District of Savannah, a city that appears three times in the whole corpus and is not a Reserve Bank. It is the best model here by every loss measure. It still cannot reproduce a twelve-row table it has read 112 times, because predicting a likely next token and recalling a fact are not the same task.

One generated minutes document from the phase 1 model, annotated. Every claim in the notes was checked against the corpus. Hover or tap a highlight.

The next step toward a model that can say what the Fed actually said is retrieval: find the real paragraph in the cleaned corpus, put it in the prompt, and let a model summarise text it can see. That is a different project with a different failure mode.

Nothing here differs in kind from what trains a frontier model: the same objective, the same transformer block, the same optimiser and schedule. Compute is estimated as 6 × parameters × tokens where not published.