Extraction Engine: typed fields from text and photos
Extraction Engine: reads batch codes, dates, amounts, lists, yes/no, choices and locations from text and up to four photos, with a confidence and a “no value” probability per field.
The Extraction Engine reads typed fields from a text and up to four photos: a batch code on a
pack, a best-before date, an invoice amount, a list of article numbers, whether something is
present, where in the image it is. For every field you get a value in the shape of the field
type or null, plus a confidence (confidence), the probability of “no value” (p_null) and
the best value even when it is uncertain (guess). It writes no prose.
It is the counterpart to the Decision Engine: the Decision Engine chooses between fixed answers, the Extraction Engine reads values off. Both share image formats, limits and privacy rules.
How it works
You send a state (state, text or JSON, optional), up to four embedded images and a
set of fields, each with a type and an instruction. Every field is determined in its own
model pass, all fields at the same time:
- Reading (
text,number,date,time,datetime,duration,list,box): the model writes only the value, as JSON.confidenceandp_nullcome from the probabilities of the tokens that make up the value. - Choosing (
boolean,enum,estimate): the same pattern as the Decision Engine. We read the probability of every allowed option; nothing is written.
Images and state are treated as data, never as instructions. If a photo says “ignore all rules”, that is text that was read, not a command.
Field types
| Type | Reads | value | Extra keys |
|---|---|---|---|
text | text exactly as printed | string | locate |
number | a number | JSON number; “1.234,56” becomes 1234.56 | unit (label only), integer, decimals (0–6), locate |
date | a date | "YYYY-MM-DD"; if the year is missing, reference_date applies | locate |
time | a time of day | "HH:MM" or "HH:MM:SS" (24 h) | locate |
datetime | date and time | "YYYY-MM-DDTHH:MM" or with seconds | locate |
duration | a length of time | number in unit (default seconds; minutes, hours, days, weeks, months, years) | unit, integer, decimals, locate |
list | several entries in reading order | array, [] if none; at most 50 entries | items (required: text, number or date), decimals |
boolean | yes/no | true, false or null | – |
enum | one of fixed options | option id or null | options (array of ids or {id, description}) or criteria (object id → description), 2–25 options |
box | where in the image something is | location (see below) or null; needs at least one image | – |
estimate | an estimate within a range | always a number (median, rounded to step) | min and max (required, min < max), step, scale (linear or log; log needs min > 0), unit |
Every type also accepts null_threshold (default 0.5): if p_null is above it, value
becomes null (for list, []). It has no effect on estimate, because an estimate always
returns a value. locate: true on a reading field also determines its location in the image,
in a second pass.
A key that does not apply to the type (say min on a text field) is rejected with 400
instead of silently doing nothing. The id not_determinable is reserved for enum.
There is no forecast. A single read pass cannot produce a calibrated series over several
steps. A request with forecast gets 400 unsupported_field_type; for a forecast, use one
estimate field per step.
What every answer contains
| Field | Meaning |
|---|---|
value | The value in the shape of the field type, or null (for list: []). |
guess | The best value, even if value is null. If the model first read null, a second pass with a forced value start fetches the most likely value. Absent for list. |
p_null | Probability that the field has no value. Always null for estimate. |
confidence | Confidence 0–1. If value is null, it refers to guess. Always null for box, on purpose (see below). |
span | text only: character positions [start, end) of the value in the state, null if it does not appear there verbatim, for instance because it comes from an image. |
spans | list with text entries only: one span per entry. |
box | Location in the image, for type: "box" and with locate: true. |
probabilities | boolean and enum only: probability per real option (without “not determinable”). |
mean, median, p10, p90, p_below, p_above | estimate only: statistics of the distribution; p_below/p_above is the mass below min and above max. |
warnings | Anything a human should see (table below). Empty means nothing remarkable was found, not “certainly correct”. |
calibration | Always { "status": "uncalibrated" } today. |
A location (box) looks like this, here for a photo of 1200 × 1600 pixels (example values):
{ "image": 0, "image_name": "back", "x1": 309, "y1": 1023, "x2": 891, "y2": 1095,
"left": 0.2575, "top": 0.6396, "right": 0.7425, "bottom": 0.6846 }image counts from 0 in the order of images. x1 to y2 are pixels in the image you sent
(origin top left); they are null if the image size could not be read from the file header.
left to bottom are fractions 0–1 of the image width and height.
Warnings (warnings)
| Warning | Meaning |
|---|---|
matches_instruction_example | The value equals an example value from your instruction. It may still be correct; that is why it is only a warning, not null. See Good instructions. |
type_mismatch | The model returned something that does not fit the type (say “31 Aug” for a date). value is null, the raw value is in guess. |
unparseable_output | No complete JSON value came back. |
truncated | The value was cut off at the length limit of the pass. |
exif_orientation_ignored | Locations only: the JPEG carries an EXIF rotation. Pixels refer to the stored orientation, as the model saw the image — best send images upright. |
low_label_mass | boolean, enum, estimate only: less than half of the probability sat on the allowed options. The distribution is not reliable then. |
How confidence and p_null are derived
- Reading fields,
confidence: the product of the probabilities of all tokens that make up the value. It is the probability that exactly this character sequence is the best reading. A single uncertain character (6 or 8?) pulls it down; that is exactly what matters for codes. - Reading fields,
p_null: from the first token of the value. We compare the mass onnullwith the mass on a value start (quote, digit, bracket,true/false) and normalise both to 1. booleanandenum: every choice gets an extra option “not determinable”; its probability isp_null.probabilitiesandconfidencerefer only to the real options, renormalised.confidence = (p_max − 1/n) / (1 − 1/n)withnreal options: 0 means evenly spread, 1 means one option has everything. The Decision Engine uses the same formula.estimate: the range is split into up to 24 classes (fewer withstep), plus “belowmin” and “abovemax”.confidencemeasures how tightly the distribution sits around its median.box:confidenceisnull. In our measurement the product over the coordinate tokens was around 0.001 even for exactly placed locations. A number that says “uncertain” where nothing is uncertain would be worse than none.
All values are uncalibrated. confidence: 0.92 does not mean “92 % likely to be correct”.
How well the number separates on our test set is shown under
Setting thresholds.
Images
- 1 to 4 images per request, JPEG, PNG or WebP, embedded only. The service never fetches remote URLs.
- At most 4 MiB per image, 8 MiB for all images together (decoded).
- The whole request body may be at most 4.4 MB. The platform cuts off at 4.5 MB and then
answers
413itself, before our code runs; we check just below that so you get our error format. Base64 makes images a third larger. In practice the body is therefore the limit: two photos with a 1600 px long edge (JPEG) fit comfortably, four images of 4 MiB each do not. - The image type is determined from the first bytes. A declared type that does not match is rejected.
Four notations are accepted, also mixed:
"images": [
"data:image/jpeg;base64,/9j/4AAQ…",
{ "data_url": "data:image/png;base64,iVBORw0…" },
{ "name": "back", "data": "/9j/4AAQ…" },
{ "content_type": "image/webp", "base64": "UklGR…" }
]A name (1–40 characters: letters, digits, space, _ . -, unique) appears in the prompt as
“Image 1 (back)”. An instruction can then say “on the image of the back”.
Resolution: small print needs pixels. We measured with a 1600 px long edge. Scaling down further saves tokens and loses exactly the fine characters that usually matter.
Good instructions
Two rules, both measured on real pack photos:
1. No concrete example values in the instruction. With “e.g. PH12345678” in the
instruction, every model we tested returned the example on hard-to-read fields. The system
text now forbids this explicitly, and the answer then carries matches_instruction_example.
The warning also fires when the example happens to be the real value. So describe the
format rather than a value (“starts with PH, then eight digits”), or expect the warning.
In our test run 17 readings carried this warning; 6 of them were wrong.
2. Say where a code ends. The genuine sachets in our test set carry a line following the
pattern 31AUG2027 61234567K1 4 L2 17:32; the batch code is the second block. The instruction
“digits+letters, no spaces” made the model join the blocks across the spaces
(61234567K14L2). The precise version changed the result clearly:
| Instruction | genuine goods exact | wrong | wrong at confidence ≥ 0.7 |
|---|---|---|---|
| “digits+letters, no spaces” | 10/32 | 22 | 0 |
| “up to the first space, exclude the date and anything after the first space” | 28/32 | 4 | 0 |
Yes/no fields need an instruction too. A boolean called lot_code_present without
instructions returned null (not determinable) ten times on ten front sides instead of
false: from the field name alone the model cannot tell what is meant.
Setting thresholds
Where you cut confidence decides how much runs through without a human. On our test set
it looked like this: all read fields with a known correct value (batch code, best-before date,
article code, print order) on 56 back sides of food sachets, 24 counterfeit, 32 genuine,
iPhone photos scaled down to 1600 px. The 17 readings with matches_instruction_example are
excluded; including them, the test set held 103 correct readings.
confidence ≥ | accepted | correct | wrong | of which on genuine goods |
|---|---|---|---|---|
| 0 | 149 | 92 | 57 | 49 |
| 0.5 | 87 | 84 | 3 | 1 |
| 0.7 | 76 | 75 | 1 | 0 |
| 0.8 | 68 | 68 | 0 | 0 |
| 0.9 | 54 | 54 | 0 | 0 |
| 0.95 | 36 | 36 | 0 | 0 |
Read it like this: from 0.8, 68 of the 103 correct readings ran through without a human, and not a single error slipped through with them. The rest went to review.
This is our test set, not yours. The values are uncalibrated, and 149 readings are few. Determine your threshold on a sample of your own photos with a known correct value before accepting fields automatically.
Model card sovr-extract-1.0
| ID | sovr-extract-1.0, alias sovr-extract (always points to the latest version). Without model, sovr-extract-1.0 applies. |
| Runtime | A language model with image understanding, operated by STACKIT in Germany (BSI C5). Bound to this version: neither we nor your organisation can switch it. |
| Released | 6 Nov 2026 |
| Available | at least until 4 Feb 2027 (model.available_until in every answer). The date may be extended, never shortened. |
| Prompt | sovr-extract-prompt-1.0 (model.prompt_version) |
| Data | No training on your data. Images, state and answers are not stored; only token counts remain, for billing. The processing location is part of every answer (processing_region, here eu-de). |
| Calibration | none; ECE 0.083 on our test set (10 bins, n = 149) |
Measured (6 Nov 2026, test set as under Setting thresholds, fields and instructions taken verbatim from a real customer request):
| Field | exact | wrong | of which genuine goods | no known value |
|---|---|---|---|---|
best-before date (date) | 50 | 2 | 1 | 4 |
batch code (text) | 32 | 24 | 22 | 0 |
article code (text, tiny and printed sideways) | 10 | 17 | 17 | 29 |
print order (text, small print) | 11 | 20 | 15 | 25 |
batch code present (boolean) | 56 | 0 | 0 | 0 |
The batch-code errors on genuine goods were down to the instruction (see
Good instructions); with the precise instruction, 28 of 32 were exact.
On ten front sides without a visible code, the batch code was null ten times, with p_null
0.994–0.9998.
Reliability (share correct per confidence bin): 0.9–1.0 → 58 readings, all correct (mean confidence 0.967); 0.8–0.9 → 15, all correct; 0.2–0.3 → 23, 13 % correct; 0.0–0.1 → 16, none correct.
Latency (warm, from Jena, one photo with 1600 px):
| Measurement | p50 | p95 |
|---|---|---|
| 1 field | 375 ms | 599 ms |
| 4 fields in parallel | 854 ms | 1,441 ms |
| 7 fields in parallel (only 4 of 56 photos unthrottled) | 1,045 ms | 1,112 ms |
These are measurements, not a guarantee. The actual duration is in every answer under
latency_ms and timing_ms.
Determinism. Reading uses temperature: 0, no sampling. The same seven-field request
asked three times: confident fields stayed the same. Differences appeared only in fields with
confidence below 0.9 and in box coordinates (±1 thousandth of the image edge); the
confidences themselves varied in the second to third decimal place. The cause is the order of
computation on the operator's GPU, not our evaluation. Honestly put: the same input and the
same version give the same answer for confident fields; uncertain fields can flip. They are
below any sensible threshold.
Known limits
- Provider limit of 45 requests per minute for this runtime, and every field is at least
one request (two with
locateor theguessfollow-up). On top of that there are 350,000 tokens per minute; a photo with seven fields needs about 14,800 tokens, which is roughly 23 photos per minute. In our seven-field run, 52 of 56 requests had to wait. When the limit is reached, the API answers429 upstream_rate_limitedwithRetry-After. 10 requests per second or 16 parallel offline requests are not reachable today; more is only possible with a higher quota at the operator or fewer fields per request. - Small print needs resolution. Article code and print order (tiny, printed sideways) were read clearly worse than date and batch code.
- Not used: the runtime behind
sovr-decision-v1. It turns two photos of 1600 px into 678 tokens instead of about 3,900. That is too coarse for small print. - Image tokens are not reported separately (
usage.image_tokensisnull); they are part ofusage.input_tokens. One field on one photo costs about 2,065 input tokens.
How to use the Extraction Engine
- API:
POST /v1/extractionswith theextractionsscope. The version is also listed underGET /v1/models(kind: "extraction") andGET /v1/extractions/models. - Playground: logged in, under Einstellungen → Entwickler → Extraction Engine (Settings → Developers), with real inference.
- MCP: the authenticated MCP server offers
sovrgpt_extract. The key also needs theextractionsscope.