๐ Evaluation
Why Should I Evaluate Models?โ
Meet Alex, a machine learning engineer at a mid-sized company. Alex knows there are numerous AI models out thereโGPTs, LLaMA, and many moreโbut which one works best for the job at hand? They all sound impressive on paper, but Alex canโt just rely on public leaderboards. These models perform differently depending on the context, and some models may have been trained on the evaluation dataset (sneaky!). Plus, the way these models write can sometimes feel โฆ off.
That's where Open WebUI comes in. It gives Alex and their team an easy way to evaluate models based on their actual needs. No convoluted math. No heavy lifting. Just thumbs up or thumbs down while interacting with the models.
TL;DRโ
- Why evaluations matter: Too many models, but not all fit your specific needs. General public leaderboards can't always be trusted.
- How to solve it: Open WebUI offers a built-in evaluation system. Use a thumbs up/down to rate model responses.
- What happens behind the scenes: Ratings adjust your personalized leaderboard, and snapshots from rated chats will be used for future model fine-tuning!
- Evaluation options:
- Arena Model: Randomly selects models for you to compare.
- Normal Interaction: Just chat like usual and rate the responses.
Why Is Public Evaluation Not Enough?โ
- Public leaderboards arenโt tailored to your specific use case.
- Some models are trained on evaluation datasets, affecting the fairness of the results.
- A model may perform well overall, but its communication style or responses just donโt fit the โvibeโ you want.
The Solution: Personalized Evaluation with Open WebUIโ
Open WebUI has a built-in evaluation feature that lets you and your team discover the model best suited for your particular needsโall while interacting with the models.
How does it work? Simple!
- During chats, leave a thumbs up if you like a response, or a thumbs down if you donโt. If the message has a sibling message (like a regenerated response or part of a side-by-side model comparison), youโre contributing to your personal leaderboard.
- Leaderboards are easily accessible in the Admin section, helping you track which models are performing best according to your team.
One cool feature? Whenever you rate a response, the system captures a snapshot of that conversation, which will later be used to refine models or even power future model training. (Do note, this is still being developed!)
Two Ways to Evaluate an AI Modelโ
Open WebUI provides two straightforward approaches for evaluating AI models.
1. Arena Modelโ
The Arena Model randomly selects from a pool of available models, making sure the evaluation is fair and unbiased. This helps in removing a potential flaw in manual comparison: ecological validity โ ensuring you donโt knowingly or unknowingly favor one model.
How to use it:
- Select a model from the Arena Model selector.
- Use it like you normally would, but now youโre in โarena mode.โ
For your feedback to affect the leaderboard, you need whatโs called a sibling message. What's a sibling message? A sibling message is just any alternative response generated by the same query (think of message regenerations or having multiple models generating responses side-by-side). This way, youโre comparing responses head-to-head.
- Scoring tip: When you thumbs up one response, the other will automatically get a thumbs down. So, be mindful and only upvote the message you believe is genuinely the best!
- Once you rate the responses, you can check out the leaderboard to see how models are stacking up.
Hereโs a sneak peek at how the Arena Model interface works:
Need more depth? You can even replicate a Chatbot Arena-style setup!
2. Normal Interactionโ
No need to switch to โarena modeโ if you don't want to. You can use Open WebUI normally and rate the AI model responses as you would in everyday operations. Just thumbs up/down the model responses, whenever you feel like it. However, if you want your feedback to be used for ranking on the leaderboard, you'll need to swap out the model and interact with a different one. This ensures there's a sibling response to compare it with โ only comparisons between two different models will influence rankings.
For instance, this is how you can rate during a normal interaction:
And here's an example of setting up a multi-model comparison, similar to an arena:
Leaderboardโ
After rating, check out the Leaderboard under the Admin Panel. This is where youโll visually see how models are performing, ranked using an Elo rating system (think chess rankings!) Youโll get a real view of which models are truly standing out during the evaluations.
This is a sample leaderboard layout:
Topic-Based Rerankingโ
When you rate chats, you can tag them by topic for more granular insights. This is especially useful if youโre working in different domains like customer service, creative writing, technical support, etc.
Automatic Taggingโ
Open WebUI tries to automatically tag chats based on the conversation topic. However, depending on the model you're using, the automatic tagging feature might sometimes fail or misinterpret the conversation. When this happens, itโs best practice to manually tag your chats to ensure the feedback is accurate.
- How to manually tag: When you rate a response, you'll have the option to add your own tags based on the conversation's context.
Don't skip this! Tagging is super powerful because it allows you to re-rank models based on specific topics. For instance, you might want to see which model performs best for answering technical support questions versus general customer inquiries.
Hereโs an example of how re-ranking looks:
Side Note: Chat Snapshots for Model Fine-Tuningโ
Whenever you rate a modelโs response, Open WebUI captures a snapshot of that chat. These snapshots can eventually be used to fine-tune your own modelsโso your evaluations feed into the continuous improvement of the AI.
(Stay tuned for more updates on this feature, it's actively being developed!)
Summaryโ
In a nutshell, Open WebUIโs evaluation system has two clear goals:
- Help you easily compare models.
- Ultimately, find the model that meshes best with your individual needs.
At its heart, the system is all about making AI model evaluation simple, transparent, and customizable for every user. Whether it's through the Arena Model or Normal Chat Interaction, youโre in full control of determining which AI model works best for your specific use case!
As always, all of your data stays securely on your instance, and nothing is shared unless you specifically opt-in for community sharing. Your privacy and data autonomy are always prioritized.