Insights
Research Notes

Why 30,000 Synthetic Profiles Are Not Proof of Accuracy

Scale makes simulation useful. Only comparison with independent human outcomes makes accuracy claims credible. Here is the evidence standard Synachron is adopting.

Synachron Team5 September 2026

Thirty thousand synthetic profiles are a scale claim. They are not an accuracy claim.

That distinction matters. A simulation can contain many internally consistent agents and still misrepresent a real audience. More responses can make a modeled distribution look precise without making it valid. If the population definition, model behavior, or calibration is wrong, scale can simply repeat the error more confidently.

At Synachron, the 30,000-plus profile library is infrastructure: reusable synthetic profiles that let teams explore scenarios, compare alternatives, and inspect modeled differences across segments. It expands the number of questions the platform can examine. It does not, by itself, prove that a result predicts what a real market, electorate, or community will do.

What would count as evidence?

First, the scenario must be frozen before the outcome is known. The population definition, model and configuration, prompt, exclusions, scoring rule, and expected result should be recorded in advance. Second, the result must be compared with independent human responses or observed behavior that the simulation could not access. Third, the comparison must report disagreement and subgroup error, not only a favorable topline correlation. Fourth, the result should remain public even when the model fails.

Research on generative agents gives good reasons to investigate this approach. Park and colleagues created interview-grounded agents for 1,052 people and reported that their agents replicated General Social Survey responses at 85 percent of the accuracy with which participants replicated their own answers two weeks later. The grounding is important: those agents were built from extensive individual interviews. The finding is not a general license to treat any demographically prompted agent as a representative respondent.

Other research reinforces both the promise and the caution. Argyle and colleagues found that language models can be conditioned to approximate patterns in human samples. Bisbee and colleagues warn that synthetic survey replacements can distort distributions and understate human variation. Santurkar and colleagues found substantial misalignment between language-model opinions and those of many demographic groups, even when models were steered toward those groups.

The practical conclusion is not that synthetic populations are useless until they perfectly predict reality. It is that their role should be explicit.

Today, Synachron is best used to pressure-test a decision: surface objections, compare directional responses, find fragile assumptions, examine segment differences, and decide what deserves the next human study, live experiment, poll, or expert review. It should not be presented as representative polling, causal evidence, or a substitute for observed behavior.

We call this evidence-gated decision simulation. Every result should carry an evidence boundary: what population was modeled, which assumptions shaped the run, which model and configuration produced it, what has been validated for this use case, and where the system should abstain.

This also changes how Synachron will build awareness. We intend to publish Prediction Receipts: timestamped simulations released before a public outcome, with assumptions and scoring rules fixed in advance, followed by an unedited assessment after reality arrives. We will publish human benchmark comparisons and failure analyses, not only successful demonstrations.

The goal is not to make synthetic minds sound certain. It is to make decision simulation reviewable, repeatable, and useful before real-world commitment.

Sources

Park et al., Generative Agent Simulations of 1,000 People: https://arxiv.org/abs/2411.10109

Argyle et al., Out of One, Many: Using Language Models to Simulate Human Samples: https://www.cambridge.org/core/journals/political-analysis/article/out-of-one-many-using-language-models-to-simulate-human-samples/035D7C8A55B237942FB6DBAD7CAA4E49

Bisbee et al., Synthetic Replacements for Human Survey Data? The Perils of Large Language Models: https://www.cambridge.org/core/journals/political-analysis/article/synthetic-replacements-for-human-survey-data-the-perils-of-large-language-models/B92267DC26195C7F36E63EA04A47D2FE

Santurkar et al., Whose Opinions Do Language Models Reflect?: https://proceedings.mlr.press/v202/santurkar23a.html

Synachron uses privacy-friendly product analytics to understand which workflows are useful. Tracking starts only after you allow it, and IP-based geolocation and session replay are disabled.

Support