Computerized Adaptive Testing

Computerized Adaptive Testing.

class torch_measure.cat.AdaptiveTester(model, strategy='fisher', n_spanning=10)[source]

Run a Computerized Adaptive Test to efficiently estimate ability.

Given a calibrated item bank (known difficulties/discriminations), adaptively selects items and updates the ability estimate after each response.

Parameters:

  • model (IRTModel) – A fitted IRT model with item parameters.

  • strategy (str or SelectionStrategy) – Item selection strategy: “fisher” (default), “spanning”, “random”, or a custom SelectionStrategy instance.

  • n_spanning (int) – Number of spanning items (only for “spanning” strategy).

run(responses, budget=None, lr=0.1, n_steps=50)[source]

Run adaptive testing on a single subject.

Parameters:

  • responses (torch.Tensor) – Full response vector for the subject (n_items,). Only items selected by the algorithm will be “seen”.

  • budget (int | None) – Maximum number of items to administer. Defaults to all items.

  • lr (float) – Learning rate for ability estimation.

  • n_steps (int) – Number of optimization steps per ability update.

Returns:

Results with: - ‘ability’: Final ability estimate (scalar) - ‘administered’: List of item indices in administration order - ‘responses’: List of responses to administered items - ‘ability_trajectory’: Ability estimate after each item

Return type:

dict

class torch_measure.cat.AnchorCalibrator(gate='global', lam_T_max=0.7, kappa=0.1, lam_bias=0.7, bias_alpha=0.5, clip_range=(0.02, 0.98), T_bounds=(0.1, 10.0))[source]

Post-hoc calibrator wrapping any logit producer.

Parameters:

  • gate ({"global", "fisher", "label_var"}) –

    How to weight the per-category fit against the pooled global fit.

    • "global": always use the global T (default).

    • "fisher": leverage-normalised Fisher information of the T-fit at T_global.

    • "label_var": ybar * (1 - ybar) over the category’s anchors.

    Effective per-category shrinkage is lam_T_max * I / (I + kappa).

  • lam_T_max (float) – Cap on the per-category shrinkage weight.

  • kappa (float) – Half-saturation of the gate. Larger kappa shrinks harder toward T_global. Ignored when gate="global".

  • lam_bias (float) – Shrinkage of per-category bias toward the pooled global bias.

  • bias_alpha (float) – Multiplier on the bias term in the final logit.

  • clip_range ((float, float)) – Output probabilities clipped to this range.

  • T_bounds ((float, float)) – Search bounds for the golden-section T fit.

Examples

>>> import torch
>>> cal = AnchorCalibrator(gate="fisher")
>>> _ = cal.fit(  # 5 anchors split across 2 categories
...     logits=torch.tensor([0.5, -0.2, 1.1, 0.0, -0.8]),
...     labels=torch.tensor([1, 0, 1, 1, 0]),
...     category=torch.tensor([0, 0, 0, 1, 1]),
... )
>>> probs = cal.transform(
...     logits=torch.tensor([0.3, -0.5]),
...     category=torch.tensor([0, 1]),
... )
fit(logits, labels, category)[source]

Fit T and bias from anchor pairs grouped by category.

Parameters:

  • logits (torch.Tensor) – Predicted logits at the anchors, shape (N,).

  • labels (torch.Tensor) – Observed binary labels at the anchors, shape (N,).

  • category (torch.Tensor) – Category index for each anchor, shape (N,).

Return type:

self

transform(logits, category)[source]

Apply the fitted calibration to held-out logits.

Categories not seen during fit fall back to (T_global, 0).

Parameters:
Return type:

Tensor

fit_transform(anchor_logits, anchor_labels, anchor_category, logits, category)[source]

Convenience: fit(anchors).transform(query).

Parameters:
Return type:

Tensor

torch_measure.cat.fisher_information(ability, difficulty, discrimination=None)[source]

Compute Fisher information for each item at given ability levels.

For the 2PL model: I(theta) = a^2 * P(theta) * (1 - P(theta)) For the Rasch model (a=1): I(theta) = P(theta) * (1 - P(theta))

Higher information means the item is more useful for estimating ability at that level.

Parameters:

  • ability (torch.Tensor) – Subject ability values, shape (N,) or scalar.

  • difficulty (torch.Tensor) – Item difficulty values, shape (M,).

  • discrimination (torch.Tensor | None) – Item discrimination values, shape (M,). Defaults to 1 (Rasch).

Returns:

Fisher information matrix, shape (N, M) or (M,) if ability is scalar.

Return type:

torch.Tensor

class torch_measure.cat.MaxInfoStrategy[source]

Select the item with maximum Fisher information at current ability estimate.

select(ability_estimate, difficulty, discrimination, administered)[source]

Select the next item to administer.

Parameters:

  • ability_estimate (torch.Tensor) – Current ability estimate (scalar).

  • difficulty (torch.Tensor) – Item difficulties (M,).

  • discrimination (torch.Tensor | None) – Item discriminations (M,).

  • administered (torch.Tensor) – Boolean mask of already-administered items (M,).

Returns:

Index of the selected item.

Return type:

int

class torch_measure.cat.RandomStrategy[source]

Select items randomly from the unadministered pool.

select(ability_estimate, difficulty, discrimination, administered)[source]

Select the next item to administer.

Parameters:

  • ability_estimate (torch.Tensor) – Current ability estimate (scalar).

  • difficulty (torch.Tensor) – Item difficulties (M,).

  • discrimination (torch.Tensor | None) – Item discriminations (M,).

  • administered (torch.Tensor) – Boolean mask of already-administered items (M,).

Returns:

Index of the selected item.

Return type:

int

class torch_measure.cat.SpanningStrategy(n_spanning=10)[source]

Select items spanning the difficulty range before switching to max-info.

First selects items evenly across the difficulty range to get a rough ability estimate, then switches to maximum information selection.

Parameters:

n_spanning (int) – Number of items to select in the spanning phase.

select(ability_estimate, difficulty, discrimination, administered)[source]

Select the next item to administer.

Parameters:

  • ability_estimate (torch.Tensor) – Current ability estimate (scalar).

  • difficulty (torch.Tensor) – Item difficulties (M,).

  • discrimination (torch.Tensor | None) – Item discriminations (M,).

  • administered (torch.Tensor) – Boolean mask of already-administered items (M,).

Returns:

Index of the selected item.

Return type:

int

reset()[source]

Reset spanning count for a new test session.

class torch_measure.cat.AdaptiveTester(model, strategy='fisher', n_spanning=10)[source]

Run a Computerized Adaptive Test to efficiently estimate ability.

Given a calibrated item bank (known difficulties/discriminations), adaptively selects items and updates the ability estimate after each response.

Parameters:

  • model (IRTModel) – A fitted IRT model with item parameters.

  • strategy (str or SelectionStrategy) – Item selection strategy: “fisher” (default), “spanning”, “random”, or a custom SelectionStrategy instance.

  • n_spanning (int) – Number of spanning items (only for “spanning” strategy).

run(responses, budget=None, lr=0.1, n_steps=50)[source]

Run adaptive testing on a single subject.

Parameters:

  • responses (torch.Tensor) – Full response vector for the subject (n_items,). Only items selected by the algorithm will be “seen”.

  • budget (int | None) – Maximum number of items to administer. Defaults to all items.

  • lr (float) – Learning rate for ability estimation.

  • n_steps (int) – Number of optimization steps per ability update.

Returns:

Results with: - ‘ability’: Final ability estimate (scalar) - ‘administered’: List of item indices in administration order - ‘responses’: List of responses to administered items - ‘ability_trajectory’: Ability estimate after each item

Return type:

dict

torch_measure.cat.fisher_information(ability, difficulty, discrimination=None)[source]

Compute Fisher information for each item at given ability levels.

For the 2PL model: I(theta) = a^2 * P(theta) * (1 - P(theta)) For the Rasch model (a=1): I(theta) = P(theta) * (1 - P(theta))

Higher information means the item is more useful for estimating ability at that level.

Parameters:

  • ability (torch.Tensor) – Subject ability values, shape (N,) or scalar.

  • difficulty (torch.Tensor) – Item difficulty values, shape (M,).

  • discrimination (torch.Tensor | None) – Item discrimination values, shape (M,). Defaults to 1 (Rasch).

Returns:

Fisher information matrix, shape (N, M) or (M,) if ability is scalar.

Return type:

torch.Tensor

class torch_measure.cat.MaxInfoStrategy[source]

Select the item with maximum Fisher information at current ability estimate.

select(ability_estimate, difficulty, discrimination, administered)[source]

Select the next item to administer.

Parameters:

  • ability_estimate (torch.Tensor) – Current ability estimate (scalar).

  • difficulty (torch.Tensor) – Item difficulties (M,).

  • discrimination (torch.Tensor | None) – Item discriminations (M,).

  • administered (torch.Tensor) – Boolean mask of already-administered items (M,).

Returns:

Index of the selected item.

Return type:

int

class torch_measure.cat.SpanningStrategy(n_spanning=10)[source]

Select items spanning the difficulty range before switching to max-info.

First selects items evenly across the difficulty range to get a rough ability estimate, then switches to maximum information selection.

Parameters:

n_spanning (int) – Number of items to select in the spanning phase.

select(ability_estimate, difficulty, discrimination, administered)[source]

Select the next item to administer.

Parameters:

  • ability_estimate (torch.Tensor) – Current ability estimate (scalar).

  • difficulty (torch.Tensor) – Item difficulties (M,).

  • discrimination (torch.Tensor | None) – Item discriminations (M,).

  • administered (torch.Tensor) – Boolean mask of already-administered items (M,).

Returns:

Index of the selected item.

Return type:

int

reset()[source]

Reset spanning count for a new test session.

class torch_measure.cat.RandomStrategy[source]

Select items randomly from the unadministered pool.

select(ability_estimate, difficulty, discrimination, administered)[source]

Select the next item to administer.

Parameters:

  • ability_estimate (torch.Tensor) – Current ability estimate (scalar).

  • difficulty (torch.Tensor) – Item difficulties (M,).

  • discrimination (torch.Tensor | None) – Item discriminations (M,).

  • administered (torch.Tensor) – Boolean mask of already-administered items (M,).

Returns:

Index of the selected item.

Return type:

int

class torch_measure.cat.AnchorCalibrator(gate='global', lam_T_max=0.7, kappa=0.1, lam_bias=0.7, bias_alpha=0.5, clip_range=(0.02, 0.98), T_bounds=(0.1, 10.0))[source]

Post-hoc calibrator wrapping any logit producer.

Parameters:

  • gate ({"global", "fisher", "label_var"}) –

    How to weight the per-category fit against the pooled global fit.

    • "global": always use the global T (default).

    • "fisher": leverage-normalised Fisher information of the T-fit at T_global.

    • "label_var": ybar * (1 - ybar) over the category’s anchors.

    Effective per-category shrinkage is lam_T_max * I / (I + kappa).

  • lam_T_max (float) – Cap on the per-category shrinkage weight.

  • kappa (float) – Half-saturation of the gate. Larger kappa shrinks harder toward T_global. Ignored when gate="global".

  • lam_bias (float) – Shrinkage of per-category bias toward the pooled global bias.

  • bias_alpha (float) – Multiplier on the bias term in the final logit.

  • clip_range ((float, float)) – Output probabilities clipped to this range.

  • T_bounds ((float, float)) – Search bounds for the golden-section T fit.

Examples

>>> import torch
>>> cal = AnchorCalibrator(gate="fisher")
>>> _ = cal.fit(  # 5 anchors split across 2 categories
...     logits=torch.tensor([0.5, -0.2, 1.1, 0.0, -0.8]),
...     labels=torch.tensor([1, 0, 1, 1, 0]),
...     category=torch.tensor([0, 0, 0, 1, 1]),
... )
>>> probs = cal.transform(
...     logits=torch.tensor([0.3, -0.5]),
...     category=torch.tensor([0, 1]),
... )
fit(logits, labels, category)[source]

Fit T and bias from anchor pairs grouped by category.

Parameters:

  • logits (torch.Tensor) – Predicted logits at the anchors, shape (N,).

  • labels (torch.Tensor) – Observed binary labels at the anchors, shape (N,).

  • category (torch.Tensor) – Category index for each anchor, shape (N,).

Return type:

self

transform(logits, category)[source]

Apply the fitted calibration to held-out logits.

Categories not seen during fit fall back to (T_global, 0).

Parameters:
Return type:

Tensor

fit_transform(anchor_logits, anchor_labels, anchor_category, logits, category)[source]

Convenience: fit(anchors).transform(query).

Parameters:
Return type:

Tensor