SovrGPT Docs

Extraction Engine: typed fields from text and photos

Extraction Engine: reads batch codes, dates, amounts, lists, yes/no, choices and locations from text and up to four photos, with a confidence and a “no value” probability per field.

The Extraction Engine reads typed fields from a text and up to four photos: a batch code on a pack, a best-before date, an invoice amount, a list of article numbers, whether something is present, where in the image it is. For every field you get a value in the shape of the field type or null, plus a confidence (confidence), the probability of “no value” (p_null) and the best value even when it is uncertain (guess). It writes no prose.

It is the counterpart to the Decision Engine: the Decision Engine chooses between fixed answers, the Extraction Engine reads values off. Both share image formats, limits and privacy rules.

How it works

You send a state (state, text or JSON, optional), up to four embedded images and a set of fields, each with a type and an instruction. Every field is determined in its own model pass, all fields at the same time:

  • Reading (text, number, date, time, datetime, duration, list, box): the model writes only the value, as JSON. confidence and p_null come from the probabilities of the tokens that make up the value.
  • Choosing (boolean, enum, estimate): the same pattern as the Decision Engine. We read the probability of every allowed option; nothing is written.

Images and state are treated as data, never as instructions. If a photo says “ignore all rules”, that is text that was read, not a command.

Field types

TypeReadsvalueExtra keys
texttext exactly as printedstringlocate
numbera numberJSON number; “1.234,56” becomes 1234.56unit (label only), integer, decimals (0–6), locate
datea date"YYYY-MM-DD"; if the year is missing, reference_date applieslocate
timea time of day"HH:MM" or "HH:MM:SS" (24 h)locate
datetimedate and time"YYYY-MM-DDTHH:MM" or with secondslocate
durationa length of timenumber in unit (default seconds; minutes, hours, days, weeks, months, years)unit, integer, decimals, locate
listseveral entries in reading orderarray, [] if none; at most 50 entriesitems (required: text, number or date), decimals
booleanyes/notrue, false or null–
enumone of fixed optionsoption id or nulloptions (array of ids or {id, description}) or criteria (object id → description), 2–25 options
boxwhere in the image something islocation (see below) or null; needs at least one image–
estimatean estimate within a rangealways a number (median, rounded to step)min and max (required, min < max), step, scale (linear or log; log needs min > 0), unit

Every type also accepts null_threshold (default 0.5): if p_null is above it, value becomes null (for list, []). It has no effect on estimate, because an estimate always returns a value. locate: true on a reading field also determines its location in the image, in a second pass.

A key that does not apply to the type (say min on a text field) is rejected with 400 instead of silently doing nothing. The id not_determinable is reserved for enum.

There is no forecast. A single read pass cannot produce a calibrated series over several steps. A request with forecast gets 400 unsupported_field_type; for a forecast, use one estimate field per step.

What every answer contains

FieldMeaning
valueThe value in the shape of the field type, or null (for list: []).
guessThe best value, even if value is null. If the model first read null, a second pass with a forced value start fetches the most likely value. Absent for list.
p_nullProbability that the field has no value. Always null for estimate.
confidenceConfidence 0–1. If value is null, it refers to guess. Always null for box, on purpose (see below).
spantext only: character positions [start, end) of the value in the state, null if it does not appear there verbatim, for instance because it comes from an image.
spanslist with text entries only: one span per entry.
boxLocation in the image, for type: "box" and with locate: true.
probabilitiesboolean and enum only: probability per real option (without “not determinable”).
mean, median, p10, p90, p_below, p_aboveestimate only: statistics of the distribution; p_below/p_above is the mass below min and above max.
warningsAnything a human should see (table below). Empty means nothing remarkable was found, not “certainly correct”.
calibrationAlways { "status": "uncalibrated" } today.

A location (box) looks like this, here for a photo of 1200 × 1600 pixels (example values):

{ "image": 0, "image_name": "back", "x1": 309, "y1": 1023, "x2": 891, "y2": 1095,
  "left": 0.2575, "top": 0.6396, "right": 0.7425, "bottom": 0.6846 }

image counts from 0 in the order of images. x1 to y2 are pixels in the image you sent (origin top left); they are null if the image size could not be read from the file header. left to bottom are fractions 0–1 of the image width and height.

Warnings (warnings)

WarningMeaning
matches_instruction_exampleThe value equals an example value from your instruction. It may still be correct; that is why it is only a warning, not null. See Good instructions.
type_mismatchThe model returned something that does not fit the type (say “31 Aug” for a date). value is null, the raw value is in guess.
unparseable_outputNo complete JSON value came back.
truncatedThe value was cut off at the length limit of the pass.
exif_orientation_ignoredLocations only: the JPEG carries an EXIF rotation. Pixels refer to the stored orientation, as the model saw the image — best send images upright.
low_label_massboolean, enum, estimate only: less than half of the probability sat on the allowed options. The distribution is not reliable then.

How confidence and p_null are derived

  • Reading fields, confidence: the product of the probabilities of all tokens that make up the value. It is the probability that exactly this character sequence is the best reading. A single uncertain character (6 or 8?) pulls it down; that is exactly what matters for codes.
  • Reading fields, p_null: from the first token of the value. We compare the mass on null with the mass on a value start (quote, digit, bracket, true/false) and normalise both to 1.
  • boolean and enum: every choice gets an extra option “not determinable”; its probability is p_null. probabilities and confidence refer only to the real options, renormalised. confidence = (p_max − 1/n) / (1 − 1/n) with n real options: 0 means evenly spread, 1 means one option has everything. The Decision Engine uses the same formula.
  • estimate: the range is split into up to 24 classes (fewer with step), plus “below min” and “above max”. confidence measures how tightly the distribution sits around its median.
  • box: confidence is null. In our measurement the product over the coordinate tokens was around 0.001 even for exactly placed locations. A number that says “uncertain” where nothing is uncertain would be worse than none.

All values are uncalibrated. confidence: 0.92 does not mean “92 % likely to be correct”. How well the number separates on our test set is shown under Setting thresholds.

Images

  • 1 to 4 images per request, JPEG, PNG or WebP, embedded only. The service never fetches remote URLs.
  • At most 4 MiB per image, 8 MiB for all images together (decoded).
  • The whole request body may be at most 4.4 MB. The platform cuts off at 4.5 MB and then answers 413 itself, before our code runs; we check just below that so you get our error format. Base64 makes images a third larger. In practice the body is therefore the limit: two photos with a 1600 px long edge (JPEG) fit comfortably, four images of 4 MiB each do not.
  • The image type is determined from the first bytes. A declared type that does not match is rejected.

Four notations are accepted, also mixed:

"images": [
  "data:image/jpeg;base64,/9j/4AAQ…",
  { "data_url": "data:image/png;base64,iVBORw0…" },
  { "name": "back", "data": "/9j/4AAQ…" },
  { "content_type": "image/webp", "base64": "UklGR…" }
]

A name (1–40 characters: letters, digits, space, _ . -, unique) appears in the prompt as “Image 1 (back)”. An instruction can then say “on the image of the back”.

Resolution: small print needs pixels. We measured with a 1600 px long edge. Scaling down further saves tokens and loses exactly the fine characters that usually matter.

Good instructions

Two rules, both measured on real pack photos:

1. No concrete example values in the instruction. With “e.g. PH12345678” in the instruction, every model we tested returned the example on hard-to-read fields. The system text now forbids this explicitly, and the answer then carries matches_instruction_example. The warning also fires when the example happens to be the real value. So describe the format rather than a value (“starts with PH, then eight digits”), or expect the warning. In our test run 17 readings carried this warning; 6 of them were wrong.

2. Say where a code ends. The genuine sachets in our test set carry a line following the pattern 31AUG2027 61234567K1 4 L2 17:32; the batch code is the second block. The instruction “digits+letters, no spaces” made the model join the blocks across the spaces (61234567K14L2). The precise version changed the result clearly:

Instructiongenuine goods exactwrongwrong at confidence ≥ 0.7
“digits+letters, no spaces”10/32220
“up to the first space, exclude the date and anything after the first space”28/3240

Yes/no fields need an instruction too. A boolean called lot_code_present without instructions returned null (not determinable) ten times on ten front sides instead of false: from the field name alone the model cannot tell what is meant.

Setting thresholds

Where you cut confidence decides how much runs through without a human. On our test set it looked like this: all read fields with a known correct value (batch code, best-before date, article code, print order) on 56 back sides of food sachets, 24 counterfeit, 32 genuine, iPhone photos scaled down to 1600 px. The 17 readings with matches_instruction_example are excluded; including them, the test set held 103 correct readings.

confidence ≥acceptedcorrectwrongof which on genuine goods
0149925749
0.5878431
0.7767510
0.8686800
0.9545400
0.95363600

Read it like this: from 0.8, 68 of the 103 correct readings ran through without a human, and not a single error slipped through with them. The rest went to review.

This is our test set, not yours. The values are uncalibrated, and 149 readings are few. Determine your threshold on a sample of your own photos with a known correct value before accepting fields automatically.

Model card sovr-extract-1.0

IDsovr-extract-1.0, alias sovr-extract (always points to the latest version). Without model, sovr-extract-1.0 applies.
RuntimeA language model with image understanding, operated by STACKIT in Germany (BSI C5). Bound to this version: neither we nor your organisation can switch it.
Released6 Nov 2026
Availableat least until 4 Feb 2027 (model.available_until in every answer). The date may be extended, never shortened.
Promptsovr-extract-prompt-1.0 (model.prompt_version)
DataNo training on your data. Images, state and answers are not stored; only token counts remain, for billing. The processing location is part of every answer (processing_region, here eu-de).
Calibrationnone; ECE 0.083 on our test set (10 bins, n = 149)

Measured (6 Nov 2026, test set as under Setting thresholds, fields and instructions taken verbatim from a real customer request):

Fieldexactwrongof which genuine goodsno known value
best-before date (date)50214
batch code (text)3224220
article code (text, tiny and printed sideways)10171729
print order (text, small print)11201525
batch code present (boolean)56000

The batch-code errors on genuine goods were down to the instruction (see Good instructions); with the precise instruction, 28 of 32 were exact. On ten front sides without a visible code, the batch code was null ten times, with p_null 0.994–0.9998.

Reliability (share correct per confidence bin): 0.9–1.0 → 58 readings, all correct (mean confidence 0.967); 0.8–0.9 → 15, all correct; 0.2–0.3 → 23, 13 % correct; 0.0–0.1 → 16, none correct.

Latency (warm, from Jena, one photo with 1600 px):

Measurementp50p95
1 field375 ms599 ms
4 fields in parallel854 ms1,441 ms
7 fields in parallel (only 4 of 56 photos unthrottled)1,045 ms1,112 ms

These are measurements, not a guarantee. The actual duration is in every answer under latency_ms and timing_ms.

Determinism. Reading uses temperature: 0, no sampling. The same seven-field request asked three times: confident fields stayed the same. Differences appeared only in fields with confidence below 0.9 and in box coordinates (±1 thousandth of the image edge); the confidences themselves varied in the second to third decimal place. The cause is the order of computation on the operator's GPU, not our evaluation. Honestly put: the same input and the same version give the same answer for confident fields; uncertain fields can flip. They are below any sensible threshold.

Known limits

  • Provider limit of 45 requests per minute for this runtime, and every field is at least one request (two with locate or the guess follow-up). On top of that there are 350,000 tokens per minute; a photo with seven fields needs about 14,800 tokens, which is roughly 23 photos per minute. In our seven-field run, 52 of 56 requests had to wait. When the limit is reached, the API answers 429 upstream_rate_limited with Retry-After. 10 requests per second or 16 parallel offline requests are not reachable today; more is only possible with a higher quota at the operator or fewer fields per request.
  • Small print needs resolution. Article code and print order (tiny, printed sideways) were read clearly worse than date and batch code.
  • Not used: the runtime behind sovr-decision-v1. It turns two photos of 1600 px into 678 tokens instead of about 3,900. That is too coarse for small print.
  • Image tokens are not reported separately (usage.image_tokens is null); they are part of usage.input_tokens. One field on one photo costs about 2,065 input tokens.

How to use the Extraction Engine

  • API: POST /v1/extractions with the extractions scope. The version is also listed under GET /v1/models (kind: "extraction") and GET /v1/extractions/models.
  • Playground: logged in, under Einstellungen → Entwickler → Extraction Engine (Settings → Developers), with real inference.
  • MCP: the authenticated MCP server offers sovrgpt_extract. The key also needs the extractions scope.
Extraction Engine: typed fields from text and photos