Calibration chartPro

A reliability diagram: predicted confidence against accuracy in bins, the diagonal, cases per bin and an expected calibration error readout.

Markdown

On this page

ProClaude Sonnet 5.5 is overconfident at the top; temperature scaling pulls it onto the diagonal.

Installation

$ pnpm dlx shadcn@latest add @chatcn-pro/calibration-chart

Usage

tsx

import { CalibrationChart } from "@/components/ui/chatcn/pro/calibration-chart"

<CalibrationChart
  title="Closed book QA"
  series={[
    { id: "raw", label: "Claude Sonnet 5.5", bins },
    { id: "scaled", label: "After scaling", bins: scaled },
  ]}
/>

Pass bins (lo, hi, mean confidence, accuracy, count), or raw predictions through binPredictions(predictions, 10). The gap between each bar and the diagonal is hatched: one angle for overconfident, the other for underconfident, so the direction reads without color. Bins under minCount cases fade and go dashed, because a bar built on nine cases should not look as certain as one built on nine hundred. A second series, for example the model after temperature scaling, draws as a line and its error sits beside the first in the readout. expectedCalibrationError and maxCalibrationError are exported for your own summaries.

Examples

States

ProUnderconfident with sparse bins, loading, empty and error.

Props

Prop	Type	Notes
`series`	`{ id, label, bins }[]`	The model first; an optional second draws as a line
`minCount`	`number`	Bins below it fade and go dashed. Default 20
`title`, `footer`	`ReactNode`	Above the readout and under the legend
`height`	`number`	Plot height in px. Default 280
`loading`	`boolean`	Axes with a static plot
`error`, `onRetry`	`ReactNode`, `() => void`	Say how to fix it

Keyboard

Key	Action
Tab	Moves through the bins; each reads its numbers

Accessibility

The plot is an image with the error in its name; every bin is a focusable button with its range, cases, confidence, accuracy and a warning when it is too small to trust.
Over and under confidence differ by hatch angle as well as hue.
The plot keeps a left to right axis in right to left layouts, as charts do.

Motion

None. The tooltip and the dimming of other bins follow the pointer instantly; the loading plot pulses only when motion is allowed.