LLM Data Matching for a Veterinary Drug Registry

Veterinary Drug Reference joins an online catalog and the state registry into one searchable dataset, with every match carrying a confidence score and rationale.

99.8% of product records imported

887 products linked to the state registry

829 matches with confidence of 0.9 or higher

Partner's Profile

Background

The client is a veterinary pharmaceuticals reference project in Ukraine. It needed a structured reference of veterinary drugs: products, packages, ATCvet classification, active substances, animal species and withdrawal periods. The data sat in two sources: about 3,200 product pages of an online catalog in HTML and an export of the Ukrainian state registry of veterinary medicines with 1,473 records. Names and manufacturers differ between the two, so they cannot be joined directly.

Objective

Turn both sources into one clean dataset, link catalog products to their state registry entries with a measurable confidence, and expose the result through a search and filter API by drug, active substance, animal species and ATCvet code. The import had to run from the admin panel as a background job, with its progress visible to the team.

Solution

We built a pipeline in four stages: HTML and CSV parsing, candidate selection, LLM matching and import. The import is an archive uploaded in the admin and processed by a background task with a progress bar. On top of the data sits a search and filter API by drug, substance, species and ATCvet. Of 3,217 product records, 3,210 were imported; the remaining seven were duplicates in the source data.

The engineering challenge was matching records that never share an exact name. Sending every product against all 1,473 registry records to an LLM would be expensive and error-prone, so a cheap deterministic filter runs first: normalization, exact, substring and fuzzy matching, plus a bonus when the brand matches the manufacturer. It keeps at most 20 candidates per product. Only then is GPT-4o-mini called, in batches of 100 items, with structured output: registration number, confidence and rationale. The model only sees a narrow candidate set, which saves tokens and reduces errors. This split of work between code and model is how we approach AI software development.

Real data needed real cleaning rules, added as we met them: for example, a country field like "FRANCE POLAND FRANCE" collapses into one country. The import itself is built for reliability, as in any database development project:

a transaction per record, so one bad row never breaks the batch
deduplication by slug
an ATCvet classification tree built with MPTT
a lock that allows only one import at a time
about 290 automated tests across the pipeline and API

Deterministic Candidate Filter

Normalization, exact, substring and fuzzy matching, plus a bonus when the brand matches the manufacturer, narrow the registry down to at most 20 candidates per product before any LLM call is made.

Batched LLM Matching with Rationale

GPT-4o-mini receives products in batches of 100 with their candidate sets and returns structured output: registration number, confidence and rationale. Of 887 linked products, 829 matches have a confidence of 0.9 or higher.

Admin Import with Progress Bar

The team uploads an archive in the admin; a background task processes it with a progress bar. Each record has its own transaction, records are deduplicated by slug, and a lock allows one import at a time. 99.8% of product records were imported.

Search and Filter API

A REST API searches and filters the reference by drug, active substance, animal species and ATCvet code. The ATCvet classification is stored as a tree, and each product carries its packages, active substances, species and withdrawal periods.