How we trained a small model to answer website questions without making things up

For the chat on our website, we don't let a language model write the answers. A small decision model picks one of the answers we have written and checked, or hands the question to a person. The model has 322 million parameters, and after one round of training it got 45 of 47 Swedish test questions right and none wrong. This post is about that training: what we started from, what we trained it on, and what we learned.

A model that picks instead of writes

Large language models such as GPT write their answers word by word. That is useful, but it means a chatbot can state opening hours or a price that nobody set. On a business's own website we wanted the opposite: only answers a person has approved.

So we use Laya, an open "System 1" model. System 1 is Kahneman's name for fast, intuitive thinking. Laya reads the visitor's question and a list of possible answers, and works out how likely each answer is. We show the most likely one when it clearly beats the runner-up, and otherwise a person takes over.

TypeSafe's Jev, released in September, works the same way but only as a hosted service. Laya's weights are open, so it runs on our own machines. It is also small: GPT-3 had 175 billion parameters, while our version of Laya has 322 million.

Swedish was the problem

We use Laya's multilingual model. On 47 Swedish test questions about our own site, it got 27 right, handed 10 to a person and got 10 wrong. Its confidence didn't help either: some wrong answers came with full confidence, often because a word in the question also appeared in the wrong answer.

The training data

We trained the model on 2,295 example questions for our own site and a few other real Swedish business sites. AI agents wrote the examples from each site's own text, without seeing the test questions. A script then removed any example that looked like a test question, because otherwise the test would measure memory rather than understanding.

Six minutes on a graphics card

Training made four passes over the examples, adjusting the model's parameters a little at a time. On a 16-core processor it took 54 minutes, and on a graphics card with 12 GB of memory about six. The result is a 640 MB file.

The results

Trained on the processor, the model got 45 of 47 right, handed 2 over and got none wrong. The run on the graphics card got 43 right and one wrong, which is within the variation we see between training runs. On 120 questions from real sites it had never seen, it went from 72 right and 36 wrong to 102 right and 10 wrong. So it isn't finished: our tests suggest the training needs three or four real sites for each kind of business.

The answers come fast. On a graphics card from 2019 in our office, an answer takes 43 milliseconds (median), and on an ordinary processor about 140.

Three things we got wrong first

• We told the model what the site was about, including the words "in Lund". It then sent 24 of 44 questions to the answer about where we are. Now it gets the question and the answers, nothing else.
• We sent it the whole chat so it would understand follow-up questions. Instead, follow-ups stuck to the previous topic. Now it sees only the latest message.
• We added a large public set of Swedish questions (MASSIVE). It made no clear difference, so we left it out.

Where the big models fit

We use a large language model once per business: it reads the website and drafts the answers, and a person checks them. After that, every visitor question goes to the small model.

Next we want to add speech. We have already tested KB-Whisper, a small Swedish speech recognition model from the National Library of Sweden, for a phone version of the chat.

Do you have questions that are really a choice between a handful of known answers, in support or booking for example? We'd be glad to test a small model on your own examples: https://oniqvision.com/build-with-us