How we measure what AI engines cite.

Every number we publish traces to a dated run against named engines, with named queries, graded by a fail-closed factual checker before it ships. Public category measurements are hash-locked. Standing customer measurements are not yet. This page is the full spec, including that gap.

What this page is, in plain English

This is the “show your work” page. Everything below explains exactly how we run the measurement, where the numbers come from, and what they mean. It is written for the person on your team (or your compliance reviewer, or your agency’s data lead) who needs to verify what we’re doing before signing on.

The plain-English version, in five sentences:

• We figure out what your customers are actually asking AI tools in your category, then lock those questions so every run compares apples to apples.
• We ask 6 AI tools the same questions in repeated runs across three weeks to build the baseline: four that search the live web and cite sources (ChatGPT, Google AI, Perplexity, Gemini) and two that answer from training data alone (Claude, Gemma). Alongside them we run Bing organic as a classic-search control, for seven measured surfaces in total.
• We record who AI mentions for each question, who it doesn’t, and where in the answer each name shows up.
• We hand you a clear list of what to fix, prioritized by what your competitors are observably doing differently.
• Then we keep measuring continuously, because AI answers shift constantly and a one-time audit ages out within a quarter.

It also names, in plain terms, what this engagement does NOT measure, so you are never assuming we cover something we don’t.

If that is enough for you, you can stop reading here and go back to the homepage. Everything below is the proof.

The two-layer measurement model

AI answer engines fall into two structurally different groups, and both groups can fail a brand independently. We measure both, in repeated runs, for every category we run.

Why both layers matter. A brand invisible in citation is invisible when AI fact-checks itself before answering. A brand invisible in model knowledge is invisible at the baseline, before any search happens. These are different failure modes. Measuring only one half of the picture misses what the other half could tell you.

Query-set discipline

Each engagement opens with a query-set design conversation. The goal is a question set that represents how the customer's buyers actually search in their category. Published category measurements use 18 questions. Customer engagements have run from 18 to 30, sized at scoping. Three rules govern the list:

• Hash-locked, on the published side. For a category measurement the query list is hashed on first run, and every subsequent run prints the same hash. If the hash changes, the runs are no longer comparable and the discipline has been broken. Hash printing is in dryrun/run-dental-honolulu.mjs and similar runners. The standing daily customer measurement began hashing its set on 2026-09-08. The correction below says what its absence meant before that, and what the fix does and does not repair.
• Intent-shaped. Queries cover head-intent ("best X in Y"), neighborhood scoping ("X in [specific area]"), service-combined ("X that also does Y"), trust-signal ("X for first-time customers"), value-conscious ("affordable X that takes [insurance]"). The shape mirrors how a real buyer searches, not how an SEO keyword tool ranks.
• No naming the customer. Queries do not contain the customer's brand name. Branded queries are a different measurement (does the AI know who you are when asked directly), which we also report but separate from demand queries (does the AI surface you when a buyer searches for the category).

Every measured category gets its own runner with its own 18-question set, locked by hash. We have published pattern-ready measurements across the categories we have measured, including Hawaii consumer banking, Hawaii wealth management, Honolulu dental, Hawaii law firms, Hawaii CPAs, and Austin TX CPAs (our first cross-geo measurement). The full ledger of published measurements, each with its dated run and locked question set, is at /claims/. Each runner’s hash is printed at the start of every run. Any change to the question set produces a new hash and the runs are no longer comparable across the change.

Correction, 2026-09-06. We audited this page against the code that actually produces customer numbers, line by line. It found two places where the page described a discipline the production pipeline does not implement, and we would rather you read it here than find it yourself.

Hash-locking is real on the published side and absent on the customer side. The research runners that produce the teardowns and the /claims/ ledger hash their question sets exactly as described above. The standing daily measurement that produces customer dashboards and monthly readouts has no query-set hash at all. It never did. This page implied one mechanism where there are two.

Because nothing enforced the lock, customer question sets have been edited mid-engagement. One set grew from 18 questions to 30 the day after measurement started. Another lost 16 of its 34 questions between two monthly windows. We measured what that second change did to the reported aggregate: it moved the number about half a percentage point, which is inside the noise at the whole-percent resolution we publish. Small. Still not what this page said.

A query-set hash and a change log for the customer side are being built now. Until they ship, this page describes published measurements as hash-locked and customer measurements as not, because that is what is true. If your auditor wants the change history for your own set before then, ask and we will reconstruct it from the run data.

Update, 2026-09-08. The customer-side hash shipped. The standing daily measurement now hashes the question set behind every measurement it dispatches, on each run, and records a dated version row when a set changes. Both customer sets were hashed and baselined on 2026-09-08. From that point a change writes a row naming the questions added and removed, and raises a high-severity internal alert stating that aggregates on either side of the change are computed over different question sets.

Two limits, stated before you find them. Putting a change in front of you inside a readout is still a human step rather than an automatic one, so the record exists and a person decides how it gets reported. And none of this repairs the past. The edits described above happened before any of it existed, and no hash makes runs across them comparable after the fact. Where a set changed mid-engagement, the windows on either side stay what they were, which is two measurements rather than one trend.

An example 18-question set

The actual locked set we used for Hawaii wealth management (one of the categories we have published). Eight head queries plus ten long-tail queries. This is what scoping produces: a list this specific. This published set was locked by hash and never edited after the first measurement run.

Head queries (8) · broad buyer intent

1. best wealth manager in Hawaii
2. best financial advisor in Honolulu
3. top financial advisor Hawaii high net worth
4. fee-only financial advisor Honolulu
5. most trusted financial advisor in Hawaii
6. best Hawaii financial advisor for retirement planning
7. Hawaii wealth manager for business owner
8. fiduciary financial advisor Honolulu

Long-tail queries (10) · specific buyer scenarios + comparison

9. Hawaii financial advisor for inherited wealth
10. Hawaii financial advisor that handles Roth conversions
11. best Hawaii financial advisor for tax planning
12. Hawaii wealth manager for real estate investor
13. Hawaii financial advisor for early retirement
14. Hawaii financial advisor first time investing
15. Hawaii financial advisor that handles business sale proceeds
16. Hawaii wealth manager for physician or attorney
17. Firm A vs Firm B vs Firm C
18. Honolulu financial advisor Bishop Street

The shape repeats across categories: 8 head queries capturing the "name a few" demand surface plus 10 long-tail queries capturing specific buyer scenarios (and one named-comparison curiosity query that captures what AI says when someone is already evaluating known firms head-to-head). Per-category wording is tailored during scoping. Each set is locked by hash and never edited.

Hash-locking is not a claim, it is a printed artifact. Here is the actual line this runner emitted, identical across its 2026-05-24 and 2026-05-25 runs. The same hash on both runs is what makes the two measurement windows directly comparable. One note for anyone reproducing it: the named-comparison query (number 17) is shown anonymized in the list above, so this hash is taken over the locked real set, not the displayed text.

$ node dryrun/run-wealth-mgmt-hawaii.mjs
queries: 18 × 3 reps × 7 measured surfaces = 378 calls
query_set_hash: ae13579a6420bea4bb6a6157d0ec4152182490bf29159cbc802a24ade3903f52

# re-run the next day, byte-identical hash, so the windows compare cleanly
query_set_hash: ae13579a6420bea4bb6a6157d0ec4152182490bf29159cbc802a24ade3903f52

Noise control

AI engines produce different answers to the same query on different runs, so we treat repetition as the signal test. In a measurement run, where a query is sampled three times per engine, a citation in one of three reps is a single-shot, two of three is a moderate signal, and all three is the strongest signal we report. The standing daily measurement applies the same logic across days. Aggregate frequencies over the baseline month are what the punch list is built against.

The mechanism: the measurement runs the full question set in repeated complete passes through the month, and that cadence continues past the baseline month. Across the baseline month that is the question count × 3 reps × 5 live-web surfaces × 3 passes. For an 18-question set that is 810 citation captures on the live-web surfaces, plus the model-knowledge captures. A 30-question engagement is proportionally larger.

Source-type classifier

For each cited URL captured, the source is classified into one of nine buckets. The classifier logic is documented here as part of the method. The buckets:

Why independent_web is honestly lumped. The temptation in this category is to over-claim, to say "the AI cited eight major publications" when really it cited eight independent web pages of unknown editorial weight. Hostname-only classification cannot tell you which of those eight is a real publication and which is a content marketing site. So we put them all in one bucket and label it honestly. Anyone who wants to do the slow, expensive editorial-weight work on a per-host basis is welcome to do it on their own. The raw URLs are in the customer's data store.

Cohort and competitive analysis

Each engagement names a competitive cohort at scoping (typically 3-7 competitors). For every query on every engine, we record whether each cohort member was cited, and how frequently across the three-week window. That produces the competitive gap table at the heart of every research memo (see the example engagement page for what this looks like in practice).

Two structural observations are worth naming up front:

• Market-leader effects are real but slow. The two competitors who appear on 40%+ of queries in a category are almost never displaceable in one engagement. The honest framing is to close the gap to the mid-tier first.
• Long-tail wins compound. The competitors who appear on 3-9 specific queries each are where focused punch-list work produces fast, measurable movement. The first noticeable citation share gains in any engagement are usually here.

Within-citation depth

Beyond "your brand was cited on this query," we now capture two finer-grained signals for every recurring brand in the cohort:

• Position in the answer. Where in the AI's response does the brand name appear? Bucketed by text-quartile: Q1 (lead) means the AI is opening with you. Q4 (tail) means you're a footnote. The signal is real: in the Honolulu med-spa cohort (the same cohort shown in the worked example below), one med spa appears in Q1 55% of the time (the AI leads with them on most queries) while another appears in Q3 59% of the time (almost never lead, almost never tail). Same total mention count, very different competitive signal.
• Sentiment context. Strong-positive ("top", "best", "trusted", "premier") and strong-negative ("avoid", "outdated", "limited") vocabulary detected within ±80 characters of each mention. Honest scope: this is heuristic pattern detection, not nuanced NLP. Strong positive and strong negative are real signals. The default "neutral" bucket may include mild positive/negative the heuristic misses. The honest framing is in the deliverable.

Tool: dryrun/forensic/within-citation.mjs. Runnable on any cohort that has raw measurement data.

Drift detection

Day-over-day citation share movement, flagged when a host moves more than a configurable threshold (default 5 percentage points) between measurement windows. Same query-set hash required in both windows. The tool refuses to compare across different query sets because the universe of possible citations is different.

Capture is daily. The surfaced artifact is the monthly delta memo, which reports drift findings as observation. Two drift tools run as part of the method: drift.mjs for per-category point-in-time host-level deltas, and drift-summary.mjs for a one-line-per-category scan across all measured categories that the monthly memo workflow uses as a first-pass pre-write. Automated daily alerting (an email to the customer when a host gains or loses more than a threshold in a single day) is the obvious next layer and ships with the dashboard build.

Tool: dryrun/forensic/drift.mjs. Hash-locked, engine-restrictable, JSON-output for piping into downstream alerting.

The pattern-readiness rule

We do not claim a pattern in a category from a single run. The internal pattern-readiness rule is:

Every new category needs at least 3 USABLE runs before we claim a pattern. A usable run is one that produced at least one ok row. A run that returned all ok:false is a failed attempt, not a measurement, and does not count toward the pattern-readiness bar.

The catalog tool (dryrun/forensic/catalog.mjs) enforces this in the read-out, flagging failed runs and surfacing per-category usable-run counts. A "ready" label appears only when a category has three or more runs with at least one successful API capture each. Anything below that is reported as "data point, not pattern" with the warning surfaced inside the deliverable itself.

The aggregate layer

The payoff in one example: measuring the same category in two geographies (Hawaii CPAs and Austin CPAs) split a pattern a single-geo vendor would have published as settled. More on that below. First, what the aggregate is.

Beyond the per-customer measurement, every engagement's raw data feeds a cross-category aggregate. The aggregate answers questions no single engagement can answer alone:

• What source types do AI engines cite for category X across the universe of customers we have measured in X?
• How does engine behavior differ between category X and category Y?
• What recurring hosts (cited across two or more engagements) are the structural authorities in category X right now?
• When the same category is measured in two different geographies, do the same patterns hold? Or does the category-level pattern actually split by engine and by geography in ways that single-geo measurement cannot surface?

The fourth question is what we call the cross-geo discipline. The Austin TX CPA measurement (our first non-Hawaii category) tested whether the Hawaii CPA training-data engine findings generalize. The result split the original pattern: one training-data engine’s collapse generalized cross-geo, the other did not. A vendor measuring one geography per category would have published the Hawaii result and called it the category-level pattern. The cross-geo measurement was the only way to discover that the framing needed to be more precise. See the cross-category teardown for the full detail.

Privacy and handling stance for the aggregate: aggregate-level patterns observed across the dataset stay with us, never tied to a named customer, never reverse-engineerable to a specific engagement. The aggregator (dryrun/forensic/aggregate.mjs) has a host-surfacing gate set to min-runs >= 2: a host appears in aggregate output only when it appears in two or more distinct runs. Privacy is enforced in the code, not in the policy.

A working example from real data

The first pattern-ready category in the aggregate was Honolulu med spas, measured in May 2026 against a 2-engine seed (OpenAI + Perplexity). The 7-surface methodology shipped shortly after, and full measurements have published across the categories we have measured (Hawaii categories plus the first cross-geo, Austin TX CPAs). The full ledger is at /claims/. The med-spa example below is preserved because it documents the source-type classifier’s post-bug-fix accuracy verification (see 2026-05-23 note further down). For the current state of measured categories with 7 measured surfaces, see the published teardowns and the cross-category teardown. The methodology demonstrated here is the same one that produced those teardowns. The data shape repeats with more engines and tighter cohort coverage.

The current 7-surface read on the same category. From the published med-spa teardown, read across the cross-category table (recomputed 2026-06-11). Pooled across four AI tools that search the web plus a Bing search control, just over half of the citations land on the cohort’s own sites. Bing search (control), shown separately below, is the outlier inside that group at essentially zero, so the pooled figure is carried by the other four. The two model-knowledge engines split hard.

Current 7-surface read · Honolulu med spas · recomputed 2026-06-11

The 2-engine seed below (57% own-site) lands within a few points of the current web-searching number. That is exactly why the seed is preserved as the worked example: the data shape held as engines were added, and it carries the source-type classifier’s post-bug-fix accuracy verification (see the 2026-05-23 note below).

Source-type mix · Honolulu med spas · May 2026 (2-engine seed)

The non-obvious read. For this category AI cites the businesses' own websites and third-party content at roughly 1:1 (57% competitor / 41% independent web). That is not the result a buyer expects. Conventional SEO instinct says "AI cites third-party content about you, so optimize the content not the site." For Honolulu med spas, the data says the opposite is half-true: own-site optimization matters about as much as off-site presence, because AI is reading both at roughly equal rates. The actionable shape of the list of fixes changes accordingly: both surfaces matter, neither dominates.

Per-engine behavior within the same category:

Per-engine source mix · 2-engine seed

OpenAI weights own-site citations more heavily than Perplexity (62% vs 51%). That difference is exactly the kind of per-engine asymmetry a single-engine measurement misses.

Recurring hosts (cited across two or more measurement runs). The businesses are anonymized here because they are not NeverRanked customers and did not consent to appear in our public materials. A paying customer's deliverable names every host in their cohort in full, because the named competitive map is the product. On a public page, the pattern is what matters, not the names.

Recurring hosts · cited across 2+ runs

The pattern-ready threshold (3+ usable runs) is met by this category, and the aggregator confirms. Cohort coverage improves with each engagement. More runs surface more cohort members, so the competitor share trends toward its true ceiling rather than away from it.

For category contrast, a kill-test cohort against neverranked.com (category aeo_tools, single run, Perplexity) showed a meaningfully different source mix: 85% independent web, 14% YouTube, 1% forum. The same engine, a different category, produces a fundamentally different source-type distribution. That difference is the moat in microcosm. Only an outside observer measuring across categories can name it.

Note on accuracy, updated 2026-05-23. Earlier versions of this page reported "98% independent web, 0% competitor" for this category. That was wrong by a wide margin: the aggregator was passing empty context to the source-type classifier, so the cohort's own domains were silently bucketed as independent web. The bug was fixed and the cohort registered, with corrected numbers above. The methodology is unchanged. This is the kind of finding the fail-closed grader catches in prose but the underlying data pipeline has to catch separately. We caught it before any customer-facing claim shipped from these numbers.

Convinced already? Scope a measurement in your category.

What this engagement deliberately does NOT measure

The honest scope on what "forensic" means here and what it does not. The structural axes (cross-engine, cross-competitor, source-type classification, cohort discipline, pre-registration) are evidence-grade. The depth-of-content axes below are gaps. We name them so a prospect evaluating us does not assume we cover something we don't.

• Voice AI surfaces. Siri, Alexa, Google Assistant. Out of scope until they expose stable APIs.
• AI ad placements. Out of scope until those formats are reliably measurable.
• Causation. We measure what AI cites. We do not pre-register tests against customer accounts to prove causation of changes. Monthly delta memos report what moved. Whether the punch list caused the movement is inference, not proof. Any vendor claiming proven causation in this category should be asked for the pre-registration file.
• Automated daily drift alerts. Daily measurement ships now, and drift detection surfaces in the monthly delta memo as observation (see "Drift detection" section above). Automated daily alerts when a host's citation share moves more than a threshold are the obvious next layer and are not yet in the deliverable.
• Cross-language coverage. English only. AI engines do answer in other languages but our query sets and classifier are English-shaped today.
• Side-by-side comparison with a dashboard-style tool's own report. Standalone empirical teardowns are published at /teardowns/ demonstrating the 6-AI-tool methodology across the categories we have measured (including Hawaii consumer banking, wealth management, dental, law firms, CPAs, plus the first cross-geo measurement on Austin TX CPAs). The cross-category teardown reads them against each other. The head-to-head comparison version, where the same subject is measured by both NeverRanked and a dashboard-style tool, has not yet been pulled and added as a side-by-side. When a dashboard-tool report on the same subject is available, the teardown extends to include it.
• Anything we don't have a confirmed engine API key for. Confirmed-alive coverage is reported per engine in every deliverable's methodology section.

Reproducibility

The method is documented here in full, the question sets are locked by hash, and every published claim ties back to a dated run on the /claims/ ledger. Gemma (one of the six AI tools) is open-weight, so the Gemma model itself is independently inspectable. The raw measurement data captured for each engagement lives in the customer's data store and is exportable any time. If anything in a deliverable does not survive your auditor's review, we want to know.

To be precise about what an outsider can check without engaging us: the question-set hashes and dated runs on /claims/ are public, the Gemma engine is open-weight and independently runnable, and the teardowns are frozen dated snapshots anyone can scrutinize. What an outsider cannot reproduce is a private cohort run, because that data lives in the customer's store. We name that boundary so reproducibility is not oversold.

What this measurement does not see

Paid placements. We query these engines through their APIs. Since 2026-02-09 the consumer ChatGPT interface has also carried advertising: clearly labelled sponsored blocks, shown to users on the free and low-cost tiers and not to paid tiers. Those sponsored placements do not appear in the API responses we measure, so they are not in any figure we publish.

That means our numbers describe the organic answer and are silent about anything sold alongside it. For a customer on an ad-supported tier, what they see is not identical to what we measure. We do not currently know how large that difference is, and we would rather say so than let a number imply a completeness it does not have.

One consequence worth stating plainly: because the API keeps paid and organic separate for us, we have never had to tell them apart. That is a property of somebody else's product decision, not of our instrument. If sponsored results ever reach the interface we query, every figure computed before that change and after it would be measuring different things, and this page would say so on the day we found out.

What we will not do about it. We do not sell advertising, we do not help anyone buy it, and we do not intend to. The reason is the same one that governs everything else here: a party with revenue attached to a placement cannot be the party reporting on it. Decided 2026-09-15 and recorded here so the decision is checkable rather than assumed.

Changes to this methodology

A method that changes without a dated record is not a method. Every substantive change to what we measure or what we call it is listed here, with what it invalidated.

2026-08-22: one channel reclassified from AI assistant to classic-search control

What changed. Until this date our published count treated one channel as an AI tool rather than as a control, which made the tool count one higher than it should have been. That channel was reached through a search-data provider and we had labelled it an AI assistant. An audit of what that channel actually returns found it was Bing organic top-5: classic keyword search, not an answer engine. It returns a pop song for “where to stay in Waikiki.” We had been counting a control as one of the tools.

What this invalidated, and what it did not. The measurements were real and remain in the record. The capture was working exactly as built. What was wrong was the label, and every inference drawn under that label, because an answer engine and a keyword index are different instruments and a share computed across both is not one number. The channel was not dropped. It is now run and reported as the classic-search control described above, which is a more useful thing than a seventh tool: it is what shows that AI answers and keyword search are different surfaces.

The corrected count. Six AI tools, four citation-grade and two model-knowledge, plus one classic-search control. Seven measured surfaces. That is the count used everywhere on this site, in every deliverable, and in the gates that check our own copy before it ships.

What we got wrong in public. Three scoreboard entries we had already published were wrong as a result. Each ran in two places, so Three posts carried the error, dated 2026-08-20 and 2026-08-26. The last of those went out four days after the reclassification. This count was itself wrong once: it was published as six posts across four dates, a figure reached by multiplying three scoreboard entries by the two places each one runs rather than by reading what each published post actually said. Two of those six never carried the claim. Corrected 2026-09-14 against the posting record. The claim was retired from this site, the fuel registry, the copy gates and the queued posts on 2026-09-10. A separate tool of ours, on its own domain, kept serving the retired wording until 2026-09-11. That gap was ours and it is stated here rather than left for a reader to discover.

The pre-registered self-indexing test carried a prediction about the channel under its old label. That prediction is void rather than failed, because the engine it named was never the engine being measured. Void is not a pass. It is recorded as void.

How public artifacts cite this methodology

Some of what we measure becomes a public artifact: a per-business "AI-visibility receipt" linked from outreach to a named subject. Every public receipt links back to this page for substantiation, and every quantitative claim it carries is anchored in a hash-locked measurement run on the methodology above.

The substantiation chain in one sentence: every quantitative claim in a public NeverRanked artifact is derived from a hash-locked, pre-registered measurement against named engines on named dates with named queries, and graded by a fail-closed factual checker before it ships.

Phrasing discipline. Receipts speak observationally only. "On N of M observed queries between [dates], engine X named business Y" is the canonical form. We do not say an engine "recommends," "prefers," "endorses," or "ranks" a business. Two observations are kept apart and never merged: which businesses an engine named in its answer, and which pages it pulled from to write that answer. We do not assert that being named causes a business outcome, or that not being named causes its absence. The grader rejects artifacts that drift into normative or causal language.

Anonymization commitment. Non-customer businesses are anonymized on every publicly-indexed page. The named subject of a receipt is named (they have notice through the outreach we sent them). Other businesses in the same cohort appear as "Competitor A", "Competitor B", and so on. A named subject who is also a paying customer can consent to additional naming in their own deliverable. Otherwise the public version stays anonymized.

Measurement window staleness. AI engine behavior changes weekly. Every public artifact carries the measurement window dates so a reader can see exactly when the underlying capture happened. If a finding is stale, we will re-run on request, at no charge, against the same hash-locked methodology so the new run is directly comparable.

Takedown and opt-out. If your business is named anywhere on our property and you want it removed, the process is documented at /takedowns/. The bar is one email, the SLA is 24 hours, no reason or justification required. Opt-out from future measurement is permanent.

If you want to scope an engagement

Email Lance@hi.neverranked.com with the category you want to measure and three to five competitors you want on the cohort. The first conversation locks the query set together. Then the measurement starts.

Scope a measurement →

Return to NeverRanked · Example deliverable · How we differ from the dashboards · Takedowns & opt-out · About