Measuring how human and artificial participants divide the work of talking — and refusing to say more than the measurement supports.
0:00–0:45 Open with the promise and the constraint in one breath: this talk is about building a real instrument for conversation, and about the discipline of not over-claiming what it reads.
Tell the room what they will leave with: a vocabulary (functional segment, quadrant, characteristic curve), an architecture, and a set of commitments they can apply to any measurement system they build.
How do human and artificial participants distribute language production, time, initiative, response, semantic contribution, repair, and influence across an evolving conversation?
Master specification · product mission
0:45–2:00 Read the sentence aloud, slowly. Every measure in the system is downstream of one of those seven nouns — that is the organising claim of the whole talk.
Point out what the sentence does not say: nothing about topic, correctness, satisfaction or learning. That absence is deliberate and it is the design.
Token counts. Latency. Thumbs up and thumbs down. Task completion. Session length. Retention. Cost per call.
Who opened the line of enquiry. Who repaired the misunderstanding, and how often. Whose idea was taken up. Whether the person is still doing the reasoning, or has handed it over.
The right-hand column is not exotic. Linguistics has described it for fifty years. What has been missing is an instrument that computes it continuously, at interaction speed, over conversations nobody transcribed by hand.
2:00–3:15 This is the motivating slide. Land the asymmetry: the left column is measured because it is easy to measure, not because it is what matters.
If the audience is industry-heavy, ask: what is your current answer to "is this assistant making our users better at their work?" Usually the honest answer is thumbs-up rate.
The rail at the bottom of every slide carries this budget. Press T to start the clock and it will tell you whether the talk is where it should be.
3:15–4:00 Keep this short. Signal that Part III is the longest and most technical, and that Part V is where the argument becomes normative — so nobody is surprised by the turn.
4:00 Dividers are for breathing and for re-orienting latecomers. Say the part title, say the question, move on within twenty seconds.
What is this conversation about? Classification, tagging, retrieval. Domain-bound: a taxonomy for physics tutoring tells you nothing about a support desk.
What was said? Abstractive compression. Useful, lossy, and unstable — two summarisers disagree and there is no ground truth to appeal to.
How was it done? Who produced what, when, in response to whom, adding what. The same measures apply to physics tutoring and to a support desk.
This system takes the third position. It is not a summariser and not a topic classifier. It reports how the work of the conversation was distributed between the participants, whatever the conversation was about.
4:20–5:40 The portability argument is the one that persuades engineers; the stability argument persuades researchers. Give both.
Anticipate the objection "surely topic matters" — yes, and topic analysis is a legitimate neighbouring discipline. The claim is only that interaction structure is separable and worth measuring on its own terms.
5:40–7:00 This is the most important conceptual move in Part I. If the audience takes one thing from this part, it should be that the message boundary is an artefact of the chat protocol, not a unit of meaning.
Note that the four levels are nested, not alternatives — measures are reported at the level where they are defined.
Good — you've got the mechanism right.positive_feedback But the oxygen comes from water, not from carbon dioxide.correction The CO₂ is fixed in the Calvin cycle instead.inform So where would you expect the labelled oxygen to show up?question
7:00–8:10 Read the utterance in a natural voice, then read the four labels. The audience should hear the seams before they see them.
Flag honestly that segmentation error propagates into every act-derived measure — this is the first appearance of a theme that runs all the way to Part V.
Portability. The same instrument reads a tutorial, a design review and a support thread.
Privacy posture. What is stored and logged is structure, not content: logs and analytics exports carry no raw text.
Claim hygiene. A label that only describes function cannot quietly become a judgement about a person.
8:10–9:20 People hear "content-agnostic" and assume "blind to the words". Correct that immediately — the distinction is between reading and adjudicating.
| Distribution | Measured as | Layer |
|---|---|---|
| Language production | Visible-token ratios: HS, AHR, ER, AR, FCR | 1 |
| Time | TTFT, generation duration, TPOT, end-to-end, human adjacent-turn latency | 1 |
| Initiative | Initiative balance over segments: initiate / respond / repair / manage | 1 |
| Response | Act-to-act transitions; ICCC Overall Responsivity | 1 · 2 |
| Semantic contribution | LCC quadrants; ICCC Newness and Communication Density | 2 |
| Repair | Repair rate over eligible turns; repair-family acts | 1 |
| Influence | ICCC Social Impact — semantic uptake by others, forward-looking | 2 |
Note the layer column. Four of the seven are computed deterministically with no model call at all. Only three require semantics — and those are the three that are optional, queued, and never enabled implicitly.
9:20–11:00 This is the map for the rest of the talk. Tell the audience they can point at any measure later and ask which row it belongs to.
The last line pre-empts the "so it's all just an LLM guessing" objection: most of the instrument is arithmetic over timestamps and tokens.
None of these is a measure of effort, contribution quality, or ownership of ideas. A falling FCR is a shorter follow-up. It is not evidence of disengagement, dependence or comprehension — those require external measures.
11:00–13:00 This is the "why this talk, why now" beat. The measures are old; the interlocutor is new, and the new interlocutor breaks the symmetry the old measures assumed.
Deliver the prohibited-interpretation box immediately after the seductive claim. That rhythm — claim, then boundary — is the deck's signature and it starts here.
13:00 Signal the register change: the next twelve minutes are conceptual, and the payoff is that Part III will make sense as engineering rather than as arbitrary formulas.
The specification operationalises these constructs; it does not cite them. This slide is a reconstruction of the lineage, offered so the formulas in Part III are legible as descendants of a literature rather than as inventions.
13:20–15:30 Do not linger on each strand; name the strand and name what it contributes. The honesty note matters — say it out loud rather than letting people read it.
If challenged on any attribution, concede readily: the point is the shape of the intellectual debt, not a citation audit.
15:30–17:10 The seventh column is the unusual one. Most metric catalogues stop at "permitted". Emphasise that the prohibited interpretation is normative and is returned inside the API response, so it travels with the number to wherever the number ends up.
unclassifiable is not a residual bin. It is returned when the evidence is inadequate — a refusal to guess. And uncertainty records how a speaker marks their certainty, not what they actually know.
17:10–19:00 Do not read all forty. Read the family names, then pick two acts the audience will not have expected — metacommunication and action_cancellation — and say why they earn a place.
The agency family is the AI-era addition: authorising, selecting and cancelling an agent's action are speech acts that barely existed in the original taxonomies.
19:00–20:40 Press Step and narrate the first three turns aloud. The moment worth waiting for is turn 3, where one tutor turn adds an acknowledgement, a correction and a question — three cells in the matrix out of one message.
Then point at the repair rate. On turn 1 it is null — no eligible human turn exists yet, so the rate is not measurable. On turn 2 it becomes 0.00, which is a measured zero. That is the whole null-is-not-zero distinction, happening on screen in two clicks.
Say the honest thing about the widget: this one computes in the browser. The real interface computes nothing and shows only what the API returned — the contrast is worth thirty seconds.
A repair rate of zero is not success.
Repair is ordinary interactional work — the machinery by which two parties keep a shared understanding. Its absence can signal disengagement as readily as it can signal that nothing needed fixing.
"Corrections went down after the model upgrade, so the model got better." Perhaps. Or users stopped checking. Or they stopped caring. Or the conversations got shorter.
Because the tempting reading is the one a dashboard invites. The metric dictionary states the prohibition and the API returns it beside the number.
19:00–20:15 This is the theoretical heart of the talk's ethics. Conversation analysis has known for decades that repair is constitutive of talk, not a defect in it.
Good place for a pause. If the room takes one prohibition home, make it this one.
Irrelevant means off the current instructional goal. Not globally meaningless, not wrong, not unimportant. A correct and insightful tangent scores as IN — and moving the goalposts, by publishing a different reference model, moves the same contribution to a different quadrant.
20:15–22:00 Draw the two axes in the air: relevance to the goal, novelty within this dialogue. Everything else in LCC is a consequence of those two questions.
The tangent example is worth telling as a story — it is the fastest way to stop people hearing "irrelevant" as "bad".
Relevance is defined relative to this document. Edit an expectation in place and every curve computed before the edit silently becomes incomparable with every curve after it — while both still carry the same name.
Versioning is what makes a curve a measurement instead of an opinion with a timestamp.
Where a learner's off-goal content does not resemble any described misconception closely enough, it is reported as unmapped and flagged — never forced into the nearest known category. An instrument that always finds something is not measuring.
22:00–23:40 This slide answers "why can't you just run LCC on any conversation?" You cannot, and the API says not_applicable rather than inventing a curve.
No instructional reference model. No notion of correctness. Every measure is referenced to the other participants and the evolving discussion — which is exactly what makes it applicable where LCC is not.
23:40–25:30 Walk the six names once. Stress the shared reference frame — that single sentence at the bottom is what distinguishes ICCC from six unrelated statistics.
Curves, not scores: a participant has a trajectory through a discussion, and the trajectory is the object of interest.
Every still-open window is recalculated when a new contribution arrives, and republished as a new version. History is never mutated: the earlier value stays readable in the ledger.
High impact is not benefit. Confusion draws responses as readily as insight does, and the measure counts responses either way.
25:30–27:00 This is a lovely engineering-meets-epistemology beat. The measure is honest about not being finished yet, and the system has a vocabulary for that state.
Contrast with the usual practice of showing a number that quietly changes when you refresh.
| Property | LCC | ICCC |
|---|---|---|
| Setting | Learner–tutor instructional dialogue | Multi-person or multi-agent discussion |
| Modelled entity | A designated learner | Every participating individual or agent |
| Reference frame | Instructional expectations, misconceptions, learner history | The other participants and the evolving discussion |
| Main measures | RN · RO · IN · IO | Participation · Responsivity · Cohesion · Impact · Newness · Density |
| Needs a reference model | Yes — or not_applicable | No |
| Needs future contributions | No | Yes, for final Social Impact |
The two engines share embeddings and semantic units, and their results may be visualised together. They are never combined arithmetically. A composite would imply a construct that neither model defines and neither has been validated for.
27:00–28:20 Product pressure always pushes toward one number. Say so explicitly, and say that resisting it is a theoretical commitment, not stubbornness.
Agreement with human annotation is not validity.
A macro-F1 of 0.8 against expert speech-act coding means the classifier reproduces expert labels reasonably well.
It says nothing about whether act frequency predicts learning, satisfaction, or group effectiveness. Those are separate empirical questions with separate evidence requirements.
ICCC formulas are reproducible operationalisations of theoretical constructs — not evidence that the curves predict anything. LCC validity depends on the quality of the reference model as much as on the classifier. Reproducible is not valid.
28:20–29:40 Close Part II here and let it sit. This sentence is the one the Part V argument will be built on, so plant it firmly.
29:40 The longest part. Warn that it goes from architecture down to specific constants, and that the constants are the point — they are what makes the theory computable.
30:00–31:30 Trace the diagram with a finger, top to bottom. The two branches under "conversation events" are the whole design in one picture: one branch is free and certain, the other is expensive and provisional.
The platform may terminate the container the moment a response is written. Any Layer 2 work left on a promise after the response would be silently lost — and the analysis would appear to succeed while being empty.
31:30–33:20 The sync/async column on the right is the thing to point at. Engineers recognise the shape immediately; researchers need the note at the bottom to see why it is forced.
HS = ΣU ÷ (ΣU+ΣC)
AHR = ΣC ÷ ΣU
ER = Ci ÷ Ui
AR = Ci ÷ Pi
FCR = Ui+1 ÷ Ci
Visible tokens only. AR is the one ratio against the full provider input, and is null unless that figure was supplied.
TTFT
generation duration
TPOT = dur ÷ (C−1)
end-to-end
HL, HL/100 tok
The first four describe the provider generating. The last describes a person. They are never summed.
lexical density
type-token ratio
moving-average TTR
sentences, questions
repeated n-gram share
Each with its method string, because a stop-list approximation and a POS tagger produce different numbers under the same name.
Each token count is labelled provider_reported, locally_counted or estimated. A missing provider figure is never silently replaced by a local estimate — the substitution would be invisible and would change every ratio above.
33:20–35:10 Move quickly through the formulas; dwell on the provenance box. It is the smallest idea on the slide and the one most often skipped in real systems.
Not reading time. Not thinking time. Not engagement, and not attention.
The field is named humanAdjacentTurnLatency precisely because that is all it is: the interval between two observable timestamps. Nothing in the measurement separates the causes.
35:10–36:50 A good slide for a joke: the long tail of this distribution is usually somebody making a cup of tea. Then land the naming point — the field name itself is a piece of governance.
36:50–38:40 Step 2 is the one to emphasise: the application does not trust the model's own account of where in the text it found something. Offsets are checked against the real string.
Step 6 matters for anyone who has run a system in mock mode and quietly shipped the numbers.
38:40–40:30 The 0.5 point is the single most misread number in similarity-based analytics. Say it twice.
"Versioned thresholds" will come back two slides from now — plant the phrase.
A contribution is never assumed to target the immediately preceding participant. Weak targeting produces weak weights rather than confident wrong answers.
cumulative · rolling_window · exponential_decay with w = e−λd. The distance unit and λ are stored with every result — a curve computed under one weighting is not comparable to one computed under another.
40:30–42:20 The ladder is the most transferable idea in this part: instead of guessing a target and proceeding confidently, resolve it, record the method, and let the confidence flow into the weight.
42:20–44:20 Frame these as epistemology expressed as engineering. Invariant 1 is the one to defend hardest: a zero standing in for an unmeasurable value corrupts every aggregate silently and permanently.
Invariant 6 is the practical one — it exists because a service that starts successfully and cannot be reached looks like a healthy deployment.
Versioned envelopes plus a pointer swap mean the value a client was given last week is still readable this week. Without that, a revisable measure like Social Impact would be unauditable — you could never reconstruct what a dashboard showed at the moment somebody made a decision from it.
An in-memory equivalent reproduces the same semantics, including generation preconditions, so the test suite exercises the real code paths rather than a stub with easier rules.
44:20–45:40 If the audience is research-heavy, compress this slide. If it is engineering-heavy, the optimistic-concurrency line is worth an extra thirty seconds.
| Target | Metrics |
|---|---|
| Speech acts | Per-class precision, recall, F1; macro-F1; accuracy; confusion matrix |
| Confidence | Expected calibration error over equal-width bins |
| Human annotation | Pairwise Cohen's κ across raters, and the mean |
| LCC | Mean absolute error per quadrant, overall MAE, dominant-quadrant accuracy against expert annotation |
| Target resolution | Exact-match accuracy and Jaccard overlap against annotated targets |
| Segmentation | Boundary precision, recall and F1 within a character tolerance |
By language, interaction type and contribution length — because an instrument that works on long English contributions may not work on short ones, or on another language.
A candidate classifier version is scored against the active one on the same gold set, and the deltas are reported with a recommendation, before anything is switched.
45:40–47:20 Note that κ requires multiple raters per item — which means gold sets are expensive, and that expense is the real barrier to instruments like this, not the modelling.
An unversioned threshold turns two different instruments into one name.
Someone nudges a relevance threshold from 0.58 to 0.55 to make a demo look better. Nothing breaks. No test fails. Six months of curves are now a mixture of two instruments, indistinguishable in the data, and every longitudinal claim built on them is void.
This is the most likely way a system like this quietly stops being a measurement system.
47:20–48:40 Tell the failure story in the right-hand card as a story. It is the most memorable thing in Part III and it costs twenty seconds.
48:40 Shift energy here — Part III was dense. This part is concrete and should move fast.
POST /threads/:id/turns
participantId, speakerType
text, submittedAt
firstTokenAt, lastTokenAt
tokenUsage + provenance
targets, clientRequestId
201 Created
layer1 measures
qualityFlags[]
versions{...}
researchIntegrityNotice
Semantic results arrive afterwards on the event stream, or on the next analysis read.
There is no batch import step and no offline pipeline in the critical path. The instrument runs at conversation speed, which is what makes the next four applications possible at all.
49:00–50:10 The researchIntegrityNotice field in the response is worth pointing at explicitly — the caveat is data, not documentation.
The API returns candidate moves, never actions. It does not send anything to a learner, does not choose, and does not adapt.
The tutoring system decides. That separation is what keeps a measurement error from becoming an intervention error, and keeps the analytics accountable to a different standard than the pedagogy.
50:10–51:40 The measurement/action separation is the most important sentence in Part IV. Say why it matters: the moment an analytics service starts acting, its errors become the learner's problem rather than the researcher's.
Whose share of the floor is above or below an equal share. Who is tracking others and who is tracking only themselves. Whose contributions get taken up.
Every low-confidence target. Every provisional impact value. Every insufficient_history null. The uncertainty is on screen, not smoothed away.
A participation grade. A ranking. A record that follows a person. These are curves about a discussion, not attributes of the people having it.
ICCC treats an agent as a participant with the same six measures. In a discussion where several assistants contribute alongside people, "who is responsive to whom" becomes a question about the system's design as much as about the group.
51:40–53:00 The third card is the ethical one and it should be delivered plainly, not defensively. Every group analytics tool drifts toward ranking unless something stops it.
| Signal | What it describes | Reads as |
|---|---|---|
| AHR, ER | How much language the assistant produces per unit of human input | Amplification |
| FCR | How much the human writes back after a completion | Follow-up compression |
| repair rate | How often the human corrects, reformulates or asks for a revision | Interaction cost |
| act transitions | Which moves follow which across the exchange | Structure |
| HL distribution | The shape of human response gaps, not the mean | Rhythm |
| initiative balance | Share of segments that initiate, respond, repair or manage | Who is driving |
Every row above describes form. None of them establishes that the assistant is helpful, correct, or good for the person using it. They are better instruments than a thumbs-up — not substitutes for asking whether the thing works.
53:00–54:20 This is the slide industry attendees will photograph. Give them a beat to do it, then read the prohibition so the photograph carries the caveat.
The distinction sounds pedantic and is not. A blanked field still tells a reader that the field exists and has a value — which is itself a disclosure, and one that invites inference about what was hidden.
The same instinct runs through the whole system: say what is true, or say nothing, but never say something shaped like an answer.
54:20–55:30 "Absent, not blanked" is a nice transferable principle. Offer it as something the audience can take back to systems that have nothing to do with conversation.
Generated sample conversations carry a synthetic-data notice, and a pattern in a simulation is a property of the generator — never evidence about people, and never pooled with real conversations.
A template fallback is always labelled as a template. Presenting one as a model-written transcript would make a broken provider look like a working instrument.
55:30–56:40 If there is a live demo, this is where it goes — scrub backwards and let the audience watch a provisional value settle. If not, describe it; the image lands either way.
56:40 Signal the register change again: from description to argument. This part contains the talk's actual opinions.
Each of those four is observable in interaction structure. Production symmetry is a ratio. Repair cost shows in repair-family acts. Initiative shows in the initiate/respond balance over segments.
This is the first period in which those quantities vary enough, across enough conversations, for the variation itself to be the object of study.
57:00–58:40 Be careful here: these are hypotheses about what varies, not findings. Say so. The claim is that the variation is measurable, not that we know its direction or its consequences.
Every one of these is a failure of the surrounding practice rather than of the arithmetic. Which is precisely why the arithmetic is not where the safeguards belong.
58:40–60:30 Number 3 is the one this field commits most often. Give a concrete example if you have one from your own domain — it makes the abstraction land.
60:30–62:00 This is the transferable thesis of the whole talk for anyone building measurement systems of any kind. Deliver the right-hand card slowly.
Until that evidence exists, these are reproducible operationalisations of theoretical constructs. That is a real and useful thing to be. It is not the same thing as a validated measure, and the system is built to keep saying so.
62:00–64:00 If there are researchers in the room, this is your recruitment slide. Say which of the five you most want collaborators on.
64:00–65:30 Read all five. This is the slide people write down. Then move to the notice and stop talking.
These measures describe observable and computationally inferred interaction patterns.
They do not, by themselves, establish learning, mastery, cognitive effort, engagement, trust, factual quality, intelligence, personality, or causal influence. Such interpretations require external measures and empirical validation.
Thank you · questions
65:30–66:00 Read the notice aloud in full. It is the only slide where reading the text verbatim is the right delivery — this is the sentence the system returns to every caller, and hearing it in a human voice is the point.
Then stop. Let the silence do the work before questions.
Leave this on screen during questions. It answers most of the "what exactly was the threshold" questions without needing to go back through the deck.