Eval platform appPro

An eval platform for one prompt on scripted fixtures: a playground with versions and a model comparison, datasets, experiments with live progress against a baseline, a review queue and the judge's rubric. Datasets start experiments, failed cases open in the playground and answers marked bad flow back into a dataset.

Markdown

On this page

ProThe returns assistant with v15 being scored against v14 right now. Watch the run finish, open the two cases it broke, load one into the playground, then label answers and add the bad ones to a dataset.

Installation

$ pnpm dlx shadcn@latest add @chatcn-pro/eval-platform-app

The template installs as a folder under components/chatcn/pro/eval-platform-app/: the app, the rail, the playground, datasets, experiments, review, scorers, shared parts and the data model.

Usage

tsx

import { EvalPlatformApp } from "@/components/chatcn/pro/eval-platform-app/eval-platform-app"

<EvalPlatformApp
  project="Returns assistant"
  promptName="returns-reply"
  versions={versions}
  playground={setup}
  models={models}
  datasets={datasets}
  experiments={experiments}
  scorers={scorers}
  rubric={rubric}
  review={review}
/>

Give it a height: it fills its parent. Dataset rows use the Pro DatasetTable shape, experiment rows the EvalResultsTable shape with each case's baseline scores, review items the ReviewQueue shape. A running experiment either streams its rows from your backend with caseCount set, or carries a live script for demos. onRunExperiment starts a run from a dataset and from Run again on a failed one.

Examples

Experiment against its baseline

v14 against v12 on the returns edge cases: seven cases improved, three regressed, sorted first. Open a case for its input, output and expected answer, then send it to the playground.

Prov14 against v12 on the returns edge cases: the summary, then every case with regressions first and a way into the playground.

Review queue

Label production answers on G, B, F and S with Z to undo. Answers marked bad or flagged go into a dataset in one step. Compare votes blind between two versions on 1 to 4; Annotate tags the exact words that broke policy.

ProLabel production answers on G, B, F and S, add the bad ones to a dataset, vote blind between two versions and tag the words that broke policy.

On a phone

Below 768 px the rail moves into a drawer, rows stack their numbers under the name and tables scroll sideways. Pass layout="mobile" to force it inside a device frame.

ProThe same app at phone width: views in a drawer, stacked rows, tables that scroll sideways and 44 px targets.

Empty, loading and viewer

A new project says how to run the first experiment. While data loads, every view holds its layout. With readOnly a viewer reads everything but cannot edit, run, promote or label.

ProA new project before its first experiment, every view loading, and a viewer who reads everything but edits nothing.

What is inside

The shell is the free ChatLayout: the rail collapses to icons with Cmd+B and shows which version production and staging serve.
Playground: the Pro PromptPlayground with variables, tools and a streamed run; Save as version adds to the Pro PromptVersions, which diffs and promotes; Compare models runs the Pro ModelCompare.
Datasets: hand-written sets edit in place in the Pro DatasetTable; sets sampled from traces are read only in the free DataTable, virtualized, so 10,000 rows scroll like ten.
Experiments: status, headline score with its change, cases and age, with a progress hairline while running. One experiment shows the Pro EvalRunProgress while it runs, the ExperimentSummary and the EvalResultsTable with regressions first.
Review: the Pro ReviewQueue, PairwiseVote and SpanAnnotator, and Add to dataset for answers marked bad.
Scorers: each scorer with its type and its latest average, then the Pro JudgeRubricEditor with a test on a sample.
While an experiment runs, a banner under the header says how far it is on every other view and leads back to it.

Props

Prop	Type	Notes
`project`, `promptName`	`string`	At the top and foot of the rail
`versions`, `deployments`, `onPromote`	`PromptVersionEntry[]`, `PromptDeployments`	Newest first; which version id each environment serves
`playground`, `models`, `tools`, `onRun`, `playgroundRun`		The playground's setup, models, tools, stream and a finished run to show first
`compare`	`EvpCompare`	Adds Compare models, with `onRun` streaming snapshots
`datasets`	`EvpDataset[]`	Large or `readOnly` sets render virtualized
`experiments`	`EvpExperiment[]`	Newest first, rows with baseline scores
`onRunExperiment`	`(request) => EvpExperiment`	Starts a run and opens it
`scorers`, `scorerInfo`	`Scorer[]`, `EvpScorerInfo[]`	Columns and what each checks
`rubric`, `judgeName`, `rubricSample`, `onTestRubric`		The judge's rubric and its test
`review`, `reviewDatasetId`	`EvpReview`, `string`	Queue, pairs, annotation
`defaultView`, `defaultExperimentId`, `defaultDatasetId`, `defaultPlaygroundTab`, `defaultReviewTab`, `defaultLabels`		Where the app opens
`loading`, `readOnly`	`boolean`	Every view loading; a viewer
`account`, `now`, `layout`		As in the other templates

Keyboard

Key	Action
`Cmd+B` or `Ctrl+B`	Collapse or expand the rail
`Cmd+/` or `Ctrl+/`	Every shortcut, grouped by area
`Up`, `Down`, `Enter`	Move through experiments or datasets and open one
`Escape`	Back to the list
`Cmd+Enter`	Run the prompt in the playground; `Escape` stops it
`G`, `B`, `F`, `S`, `Z`	Label good, bad, flag, skip, undo
`1` to `4`	Vote A, B, tie or both bad

Accessibility

Landmarks from the layout, a skip link, a named navigation with aria-current, and one heading level 1 per view.
Status never rests on color: running, finished and failed pair their hue with a glyph and a word, score changes with an arrow, a sign and a word.
Each experiment and dataset row is one button whose name reads its status, progress and age.
A failed run is an alert that says how to fix it. Starting and finishing runs, saving versions and adding answers to a dataset are announced politely.

Motion

The running hairline and progress dots move on short transitions; review cards enter on the popover curve; the playground streams with no per word effect.
Moving between views, lists and detail pages is instant: it is used all day and from the keyboard.
Reduced motion is handled by each composed part.