Human Runtime Cognitive Metrics

Human Runtime currently has a useful conceptual vocabulary:

• perceptual throughput;
• context utilization;
• cognitive or context headroom;
• control-loop coherence;
• goal or attention alignment;
• retrieval latency;
• context pollution;
• anticipation horizon;
• recovery latency.

These terms describe important aspects of cognition in operation. They do not yet constitute an observability metric system comparable to the one used for software and LLMs.

Datadog does not begin with a single “model health” score. It collects durations, counts, errors, tokens, cache activity, tool calls, trace structure, and outcomes. Higher-level conclusions are built from those records.

HRT needs the same separation.

The protocol objects describe where a human-runtime record belongs. The metrics describe what happened inside that record.

hrt.node, hrt.state, hrt.span, hrt.event, and hrt.trace provide the structure. A cognitive metric vocabulary supplies the measurable content.

The missing layer

The present HRT vocabulary includes five higher-level operational views:

• Context Headroom
• Attention Alignment
• Inference Stability
• Operator Automaticity
• Runtime Load

These are HRT synthesis labels, not five newly discovered cognitive faculties and not five universally accepted scientific measures. They are closer to an observability platform’s anomaly, health, or quality views than to its primary telemetry. An agent interprets them through their source, window, method, uncertainty, and validation evidence.

How these views relate to established constructs

The underlying ideas have strong precedents, even where HRT uses a new label:

The mapping matters because the literature does not support collapsing these constructs into one latent “cognitive state.” NASA-TLX treats workload as multidimensional.¹ ISO 17488 uses a detection-response task to assess attentional effects of cognitive load.² Work on spare cognitive capacity uses secondary-task performance as a workload-sensitive probe.³ Automaticity studies commonly use reduced dual-task cost.⁴ Evidence-accumulation and within-person variability research provide better foundations for stability than a free-standing “inference stability” score.⁵⁶

HRT therefore treats the five names as views assembled from lower-level records. A provider can implement and validate a proprietary estimate, but the protocol identifies it as provider-derived rather than presenting the label itself as scientific validation.

HRT therefore needs four explicit metric classes.

1. Observed primitives

Events and quantities directly recorded by a device or task system:

• cue became available;
• first fixation occurred;
• fixation duration;
• gaze transition;
• control input;
• response timestamp;
• correction;
• task error;
• pupil change;
• heart-rate or other physiological observation.

2. Operational derived metrics

Transparent calculations over observed events:

• time to first relevant fixation;
• missed-critical-cue rate;
• relevant-attention share;
• context-switch rate;
• decision latency;
• correction latency;
• recovery latency;
• cue-to-action throughput;
• revisit burden.

3. Provider-derived states

Estimates produced by proprietary or learned models:

• residual capacity;
• concentration;
• operator automaticity;
• cognitive load;
• execution flow;
• overload risk;
• fatigue or readiness estimate.

These may remain black-box provider outputs. They carry provider identity, an opaque method-version identifier, validity conditions, and enough quality or uncertainty information for safe use. HRT does not require the provider to disclose its formula, features, thresholds, weights, calibration dataset, or internal confidence function.

4. Evaluations and outcomes

Evidence about whether the runtime episode worked:

• decision correctness;
• instructor or policy grade;
• task success;
• time or safety delta;
• intervention effect;
• next-attempt improvement;
• later retention or transfer.

The four classes are never silently collapsed into one score.

The LLM-to-human metric translation

The analogy is operational rather than biological. Humans do not process literal tokens, and HRT does not invent a universal “human token.” It defines measurable equivalents around task-relevant information.

The analogy is not a claim that cognition and transformer inference use the same mechanism. It is a design guide for creating similarly useful operational observability.

Core metric families

1. Context supply

Before measuring cognition, HRT must describe the information environment.

Candidate records:

• hrt.context.cue.available
• hrt.context.cue.critical
• hrt.context.cue.expired
• hrt.context.cue.changed
• hrt.context.available.count
• hrt.context.available.relevance_weight
• hrt.context.arrival_rate

A cue can be an instrument state, warning, visual landmark, agent recommendation, spoken instruction, memory request, or other task-relevant unit.

The task or vertical profile—not HRT alone—defines cue identity, relevance, criticality, and validity window.

This is the equivalent of recording how many tokens or messages entered an LLM call. Without it, context utilization cannot be calculated.

2. Context acquisition and utilization

Candidate metrics:

• hrt.context.sampled.count
• hrt.context.sampled_relevant.count
• hrt.context.missed_critical.count
• hrt.context.utilization
• hrt.context.pollution
• hrt.context.persistence
• hrt.context.revisit_burden

Possible transparent formulas:

context utilization = relevant cue weight sampled within its useful window
                      / relevant cue weight available

context pollution = irrelevant or expired sampling time
                    / total sampling time

critical cue recall = critical cues sampled within their useful window
                      / critical cues available

attention precision = relevant sampling time
                      / total sampling time

revisit burden = repeated inspections required before correct action
                 / successfully incorporated cues

“Sampled” does not automatically mean “understood.” When comprehension is not directly testable, the protocol says visibly_sampled or interaction_sampled, not incorporated.

3. Latency

Latency is the strongest immediate bridge between software, LLM, and cognitive observability.

Candidate metrics:

• hrt.latency.cue_to_orient_ms
• hrt.latency.cue_to_first_fixation_ms
• hrt.latency.fixation_to_action_ms
• hrt.latency.cue_to_decision_ms
• hrt.latency.cue_to_action_ms
• hrt.latency.context_sync_ms
• hrt.latency.correction_ms
• hrt.latency.recovery_ms
• hrt.latency.resumption_ms

These metrics identify both endpoints. “Reaction time” alone is too vague. Cue appearance to first fixation, first fixation to action, and error to correction describe different failure stages.

4. Throughput

Perceptual throughput is a family of rates rather than one unexplained score.

Candidate metrics:

• hrt.throughput.relevant_cues_sampled_per_min
• hrt.throughput.critical_cues_sampled_per_min
• hrt.throughput.correct_actions_per_min
• hrt.throughput.verified_decisions_per_min
• hrt.throughput.successful_loops_per_min
• hrt.throughput.task_information_rate

The strongest general metric is not total eye movement or total cues seen. It is successful task-relevant perception–action loops per unit time.

effective throughput = correctly resolved task-relevant cue weight
                       / observation window

The cue weighting and correctness rule must come from the vertical profile.

5. Selection and alignment

Attention Alignment and Goal Alignment are decomposable views.

Candidate metrics:

• hrt.attention.relevant_share
• hrt.attention.critical_recall
• hrt.attention.precision
• hrt.attention.transition_rate
• hrt.attention.switch_cost_ms
• hrt.attention.entropy
• hrt.attention.tunneling_duration_ms
• hrt.attention.goal_alignment

Goal alignment can be derived from the time-weighted relevance of sampled cues to the active objective:

goal alignment = sum(sample duration × cue relevance to active goal)
                 / total sample duration

This is only meaningful when the task model identifies the active goal and cue relevance.

6. Control-loop coherence

This family measures whether perception, decision, and action remain connected in the correct order.

Candidate metrics:

• hrt.loop.completed.count
• hrt.loop.broken.count
• hrt.loop.look_without_action_rate
• hrt.loop.action_without_recent_sampling_rate
• hrt.loop.correction_rate
• hrt.loop.sequence_deviation
• hrt.loop.temporal_variability
• hrt.loop.coherence

A loop may be defined as:

cue available → relevant sample → decision or commitment → action → outcome

Different verticals can define expected sequences without changing the common HRT structure.

7. Anticipation

Anticipation Horizon uses task-native units.

Candidate metrics:

• hrt.anticipation.horizon_ms
• hrt.anticipation.horizon_m
• hrt.anticipation.future_cue_share
• hrt.anticipation.late_acquisition.count
• hrt.anticipation.reference_delta_ms

In racing, the horizon may be seconds or metres ahead of the vehicle. In aviation, it may be time before a mode transition or conflict. In a software-review workflow, it may be the number of steps before an expected consequence.

8. Mobilization and cognitive cost

The human analogue of compute usage and cost needs multiple signals.

Candidate metrics:

• hrt.mobilization.pupil_response
• hrt.mobilization.autonomic_response
• hrt.mobilization.provider_score
• hrt.cost.time_per_success_ms
• hrt.cost.fixations_per_success
• hrt.cost.corrections_per_success
• hrt.efficiency.outcome_per_mobilization
• hrt.efficiency.performance_per_attention_time

Pupil or physiological change is not direct energy consumption. It can contribute to a calibrated mobilization estimate.

mobilization efficiency = successful outcome value
                          / calibrated mobilization estimate

The numerator and denominator must be retained separately so the ratio can be audited.

9. Capacity, saturation, and headroom

Headroom is not calculated as 1 - workload or 100 - cognitive_load_score unless a provider has validated that relationship.

The more defensible definition is:

Context Headroom is the predicted additional task demand that can be introduced before a defined performance, safety, or stability boundary is crossed.

Candidate metrics:

• hrt.capacity.current_demand
• hrt.capacity.demonstrated_limit
• hrt.capacity.headroom
• hrt.capacity.recruitment_slope
• hrt.capacity.saturation_onset
• hrt.capacity.collapse_point
• hrt.capacity.forecast_horizon_ms

These require a demand model and calibration protocol.

headroom = predicted demand at failure boundary - current task demand

The unit can be concurrent targets, events per minute, task-difficulty level, interruption rate, or another vertical-specific quantity. A normalized ratio may be published in addition, while the task-native quantity remains available.

10. Recovery and resilience

Candidate metrics:

• hrt.recovery.orienting_ms
• hrt.recovery.control_ms
• hrt.recovery.baseline_ms
• hrt.recovery.error_recurrence_rate
• hrt.recovery.success_rate
• hrt.resilience.performance_drop
• hrt.resilience.recovery_area

Recovery is not one timestamp. HRT distinguishes noticing the disturbance, restoring effective action, and returning to a stable reference pattern.

11. Supervision and human–agent coordination

For human-in-the-middle systems, HRT also needs joint metrics:

• hrt.oversight.actions_per_verification
• hrt.oversight.evidence_inspected_share
• hrt.oversight.catch_rate
• hrt.oversight.false_override_rate
• hrt.oversight.approval_latency_ms
• hrt.oversight.context_sync_ms
• hrt.oversight.bandwidth
• hrt.oversight.integrity
• hrt.trust.calibration_error

These describe whether oversight was effective rather than merely present.

A shared metric vocabulary with a provider-controlled implementation

A metric contract standardizes what an output means without standardizing how a provider produces it. hrt.capacity.headroom, for example, has a shared semantic meaning while a specialist provider controls its signal processing, feature engineering, calibration, thresholds, model, and disclosure choices.

The open HRT layer defines:

• the semantic meaning and unit of a metric;
• the task, person, span, and time window to which it applies;
• whether it was observed, transparently derived, provider-estimated, or validated as a forecast;
• its permitted uses and validity conditions;
• an opaque provider and method-version identifier;
• sufficient quality or uncertainty information to prevent unsafe interpretation.

It does not require a proprietary provider to disclose:

• raw eye-tracking or biometric streams;
• internal features or intermediate values;
• model weights, formulas, decision thresholds, or source code;
• training or calibration datasets;
• detailed test vectors that reveal boundary behavior;
• the internal mapping from raw signals to confidence or state.

Metric names do not disclose the processor. Large collections of high-resolution, synchronized input–output pairs can, however, reveal aspects of its behavior. HRT can make that exposure a deliberate provider choice by supporting disclosure tiers, query scope, precision controls, retention rules, and separate permissions for comparison, learning, and joint development.

HRT therefore supports a protected provider boundary:

1. Raw signals are processed on-device or inside the provider-controlled service.
2. HRT receives only the minimum output required for the authorized task.
3. Provider estimates may be categorical, quantized, windowed, delayed, or rate-limited when exact values are unnecessary.
4. Quality can be expressed as a usable band or status rather than an exact internal confidence value.
5. Access is scoped by purpose, subject, metric, resolution, frequency, retention period, and receiving agent.
6. Adaptive or bulk queries are monitored, budgeted, and revocable.
7. Input data and provider outputs are not made jointly exportable by default.
8. Logs record which agent accessed which result and why.
9. Agreements declare which forms of learning, validation, benchmarking, and model development are permitted at that access tier. A provider may authorize deeper access for selected research or development partners without granting the same access to every network participant.

Technical controls make the declared disclosure tier enforceable. Agreements and machine-readable permissions make the intended relationship explicit. The two work together.

A graduated disclosure model

HRT does not treat learning how a provider works as inherently prohibited. A provider may benefit from allowing selected partners to study, validate, extend, or jointly develop parts of its method. The protocol makes that choice deliberate rather than forcing a binary choice between complete secrecy and complete disclosure.

A provider could offer several access tiers:

• Result access — an agent receives only a signed state, category, or recommendation.
• Metric access — an agent receives bounded values, uncertainty bands, and provenance, but no raw inputs or intermediate features.
• Validation access — a partner can run approved datasets and inspect evaluation results inside a controlled environment.
• Research access — selected researchers receive specified signals, features, or explanations under a purpose-specific agreement.
• Development access — a joint-development partner can inspect or modify agreed parts of the processing pipeline.

Each result can carry a machine-readable disclosure policy identifying the allowed tier, purpose, recipient, retention period, onward-sharing rules, and whether the data may be used for evaluation or learning.

Pluggable processors must also be comparable

Selective disclosure does not become protection from legitimate comparison. HRT supports more than one cognitive processor. An integrator can discover several compatible providers, route an authorized episode to one or more of them, and evaluate which processor works best for a particular task, population, device configuration, latency requirement, or operating environment.

The shared processor contract describes:

• processor identity and version;
• supported HRT metric semantics;
• required input modalities and quality conditions;
• supported task profiles and declared validity populations;
• deployment modes, latency expectations, and availability;
• output units, ranges, quality representation, and abstention behavior;
• evidence maturity and known limitations;
• permitted comparison, retention, learning, and disclosure tiers.

The contract makes processors substitutable at the interface, not identical in capability. Providers may support a common metric such as cue-to-action latency while also emitting differentiated metrics in their own namespaces.

HRT supports several forms of comparison:

1. Conformance comparison — does the processor accept and emit valid HRT records?
2. Operational comparison — how do latency, availability, coverage, abstention, and resource requirements differ?
3. Outcome comparison — how well do the estimates predict or improve an independently measured task outcome?
4. Calibration comparison — do stated quality or risk bands correspond to observed frequencies under declared conditions?
5. Robustness comparison — how does performance change across devices, environments, populations, missing signals, and distribution shifts?
6. System comparison — what happens to the full human–agent workflow when the processor is used for routing, timing, intervention, or supervision?

The strongest benchmark uses independent outcomes rather than treating one provider's output as ground truth. Two proprietary provider scores may not be numerically interchangeable even when their labels sound similar. HRT compares both against a declared task outcome, reference annotation, or intervention effect, and preserves the conditions under which the comparison was made.

A common evaluation trace can correlate:

authorized input episode
    → processor A output
    → processor B output
    → later task outcome or expert evaluation

The same episode may be processed in shadow mode so that neither output changes the task. This permits fair head-to-head evaluation without exposing either implementation. Results remain multidimensional and task-specific; HRT does not manufacture a universal processor leaderboard from incomparable deployments.

Agent-to-agent infrastructure can carry discovery, negotiation, task requests, and results. A processor can advertise its HRT capability profile and accept an authorized evaluation task through A2A or another agent transport. HRT still needs to define the cognitive metric schemas, evaluation records, authorization fields, and comparison semantics inside those messages. The transport moves the request; HRT makes the result comparable.

What technical protection can and cannot do

The most practical protected implementation is a provider-controlled processor. Raw data enters a provider-hosted service or a provider-controlled local module; the HRT network receives signed outputs at the authorized disclosure tier. This keeps the method and intermediate data behind a defined interface while still allowing authorized evaluation and comparison.

Encryption protects data while it is moving or stored. It does not stop an authorized receiving agent from inspecting data after decryption. Useful patterns include transport encryption, encrypted storage, provider-controlled keys, short-lived access tokens, and envelope encryption in which only the intended processor can open the payload.

A sealed local processor can keep computation near the device. A signed binary, container, or WebAssembly module can make integration easy, but software obfuscation alone only increases the cost of inspection and is not presented as a guarantee. A trusted execution environment with remote attestation can offer a stronger boundary by running the processor in isolated memory and releasing only approved outputs, although deployment complexity and platform dependence increase.

For controlled research, a data clean room or provider-hosted sandbox can allow approved analysis without allowing the underlying dataset or full input–output history to be exported. Query budgets, result aggregation, output review, and audit logs help a provider allow learning while limiting bulk extraction.

More specialized cryptographic techniques can serve narrower cases. Secure multi-party computation can calculate agreed joint statistics without either organization disclosing all of its source data. Fully homomorphic encryption can in principle compute on encrypted inputs, but it is unlikely to be the first practical choice for a latency-sensitive, evolving cognitive pipeline. Differential privacy can protect people in aggregate research outputs, but it is generally not a replacement for accurate individual runtime estimates.

The design objective is therefore not perfect secrecy. It is selective legibility: expose enough for an agent or partner to use, evaluate, and sometimes learn from a specialist capability, while letting its provider control which layer becomes visible to whom.

HRT conformance tests the interface; it does not reproduce the processor. A provider demonstrates that it emits a valid record, uses the declared unit, respects authorization, and behaves within published validity conditions without passing a public suite that contains enough labeled raw inputs and expected proprietary outputs to reconstruct its method.

At the default integration tier, this produces a deliberate asymmetry: agents know how to interpret and route a provider result without learning how the provider calculated it. The provider can open selected layers for validation, research, or joint development when the relationship justifies it. Interoperability does not make the processor an open implementation, but it also does not prevent a provider from teaching trusted partners.

Metric maturity levels

HRT does not present every candidate as equally established.

Level A — observed

Direct events and durations with clear instrumentation: fixations, cue availability, action timestamps, control inputs, errors, and outcomes.

Level B — transparent derived

Auditable formulas over Level A records: cue-to-fixation latency, relevant-attention share, missed-cue rate, correction rate, and throughput.

Level C — provider estimate

A model-based measure such as residual capacity, concentration, load, or automaticity. It requires provider, an opaque version identifier, validity metadata, and a declared quality representation. That representation may be an exact confidence value, an uncertainty band, or a simpler usability status appropriate to the disclosure tier.

Level D — validated forecast

A prospective claim that predicts future failure, remaining capacity, recovery, or intervention effect. It requires a declared validation population, outcome, horizon, and error characteristics.

This maturity ladder is analogous to separating a raw span duration from an anomaly classification or model-quality evaluation.

Every metric needs a measurement contract

A metric name and number are insufficient. Every HRT metric can carry:

name: hrt.latency.cue_to_action_ms
class: transparent_derived
value: 385
unit: ms
window:
  start: 12.080
  end: 12.465
subject: hrt.node.driver_07
span: hrt.span.corner_07
provider: hrt-reference-calculator
method_version: calc-v0.1.0 # an opaque identifier is sufficient for proprietary methods
sources:
  - task.cue.braking_marker.visible
  - vehicle.brake.onset
quality:
  clock_uncertainty_ms: 8
  cue_visibility_confidence: 0.97
validity:
  task_profile: racing.corner-entry.v1
authorization:
  purpose: training
  expires_at: 2026-12-31T00:00:00Z

Provider-derived states add calibration, model version, confidence, and supported-use fields. Evaluations add evaluator identity, rubric, and outcome evidence.

A minimum viable HRT metric set

The first interoperable version remains small. It prioritizes records that can be reproduced without accepting one provider’s model.

Universal event primitives

• cue available;
• cue visibly sampled;
• action initiated;
• action completed;
• correction initiated;
• task outcome recorded;
• interruption began and ended.

Universal durations

• cue to first relevant sample;
• relevant sample to action;
• cue to action;
• error to correction;
• interruption end to effective resumption.

Task-profile ratios

• critical cue recall;
• relevant attention share;
• missed-critical-cue rate;
• correction rate;
• successful loop rate.

Provider extension boundary

• load estimate;
• residual-capacity estimate;
• concentration or focus estimate;
• automaticity estimate;
• execution-flow estimate.

This gives HRT a measurable substrate immediately while allowing specialist providers to supply more sophisticated analytics.

What this establishes for the HRT operating model

HRT does not claim that hrt.state itself is the product. A state object without a metric surface is only a container.

The stronger formulation is:

HRT provides a common trace structure and metric contract for human cognition in operation. Devices record events. Providers derive specialized states. Vertical profiles define task relevance and outcomes. Agents consume authorized metrics. Evaluations determine what actually worked.

This creates distinct roles:

• device companies provide observable signals;
• specialist analytics providers provide proprietary cognitive analytics;
• HRT supplies the interoperable event, metric, and trace contracts;
• vertical partners define cues, goals, limits, and outcomes;
• observability platforms store, query, visualize, and evaluate the resulting telemetry;
• agents use bounded context to coordinate with the human node.

The protocol becomes more than a vocabulary for nodes and spans. It becomes the shared measurement surface that lets the ecosystem compare, combine, and validate cognition-related data without pretending that every value is directly observed or universally meaningful.

Bottom line

The earlier cognitive vocabulary was pointing in the right direction. The gap was granularity.

Perceptual Throughput, Context Headroom, Attention Alignment, Inference Stability, Operator Automaticity, and Runtime Load are top-level operational views. Beneath them, HRT uses the equivalent of tokens, durations, cache activity, errors, and evaluation results:

• cues available and sampled;
• critical cues missed;
• time to orient, decide, act, correct, and recover;
• relevant information processed per unit time;
• attention spent on relevant versus irrelevant context;
• completed and broken perception–action loops;
• mobilization cost per successful outcome;
• additional demand sustainable before failure;
• outcome-linked evaluations and intervention effects.

Datadog can show a model’s runtime because it has both traces and metrics. HRT reaches cognitive observability when its human traces carry equally explicit operational measurements.

Related article: Competition and Composability Between Processors While Protecting Their Implementations

Notes

1. NASA Task Load Index — multidimensional subjective workload across mental, physical, temporal, performance, effort, and frustration dimensions.
2. ISO 17488 detection-response task — a standardized method for assessing attentional effects of cognitive load in driving.
3. Real-time prediction of short-timescale fluctuations in cognitive workload — uses detection-response performance as a proxy for rapidly changing spare cognitive capacity.
4. The neural correlates of motor skill automaticity — operationalizes automaticity through diminishing dual-task cost after training.
5. Evidence accumulation modelling in the wild — connects evidence-accumulation models to safety-critical decisions in applied domains.
6. Cognitive performance inconsistency: intraindividual change and variability — treats within-person inconsistency as distinct from average ability and shows that it changes with practice.
7. Limitations on the Use of Eye-Tracking Data to Understand Operator Awareness — cautions against treating gaze as equivalent to awareness or understanding.