Benchmarking to show me I'm wrong

Something suspicious was going on in an embedding application of mine; large sections of text were processing almost as fast as smaller ones. I ran a benchmark, and lo and behold; any note with text beyond 5 paragraphs would just flatline in processing times. That's odd... I traced it back to the main embedding pipeline text -> [embedder] -> embedding. Turns out, the pipeline was taking only the first 5 paragraphs and disregarded the rest--without any warning. All the data I was doing similarity matches on was only taking the first few paragraphs... oops.

This wasn't a bug. It was a false assumption in the most important part of my application. I stumbled on this bug, but what I learned since then is that benchmarking is an absolute requirement when building something that needs to work well. A surprising amount of times the benchmarks have shown me that;

• I misunderstood the inner-workings.
• I had an inaccurate view of the data landscape.
• or there were nuances/edge-cases I had never considered.

What was wrong with my embedder?

So yeah, the text -> [embedder] -> embedding shape does not mean the output embedding entails all the text that was input. I knew AI models had limited input tokens and that an input would need to be chunked. But I had assumed the huggingface software was taking care of that in the background, especially since the code looked like pipeline(text, { pooling: 'mean' }) --implying to me it was mean pooling the chunks it created for itself.

Mean pooling wasn't on the implicit chunks, it was a way to explicitly tell the pipeline what algorithm to use to turn the model output into one clean vector/embedding. Aaannd, huggingface definitely was not chunking things for me, if the input was too large, it'd just use whatever it could and toss the rest. Now, do I think an invalid input warrants an error--or a warning at the very least--yes, but I really don't think that was the issue. I simply misunderstood and didn't measure what was important to me; the quality of the similarity search.

Other examples

So as I started benchmarking more, I got to face more and more misconceptions, and learned a lot by just reviewing the data. Some other interesting findings through benchmarking this project;

• Smaller AI models does not mean faster. (smaller quantized embedding models are actually slower and less optimized for the hardware)
• Cosine similarity search accuracy degradation is almost negligible when going from float32 to uint8 (so--pretty much free lunch in processing and storage)
• JSON (de)serialization is slow af--especially compared to binary storage. (JSON known for being hyper-optimized, but still slow unfortunately)
• English or multi-lingual embedding models have a lot of nuance. (English models can still be somewhat valuable in some other languages, and suffer in others)

Now with AI coding agents, these benchmarks are essentially free too. I can create and spin up a in-app benchmark or a jupyter notebook in a matter of minutes. It gives so much insight, for almost no cost in time and resources.

Bonus benefit: being able to say 'Today, I verifiably made this 70% faster' is very motivating.

Conclusion

This last year, I probably ran like a dozen benchmarks surrounding my embedding software--and it shows. Today, the Obsidian Similarity plugin is literally 20 times faster and 10 times smaller than it was when I shipped it--all while quality has improved. That's not a flex on some kind of genius design (though there were clever tricks I've done), it's more a testament on how many things I misunderstood that were raised to my attention through benchmarks and proper measurements.

As I, and many others, are pushing towards things that are less understood, the question should not end with "is it working?" or even just "is it built right?"--it's "can I afford being wrong about this?", if the answer is no, then proper benchmarking has to follow.