Hyper Speed: When Big Data Blooms – Scott Helme – NDC Security 2022

Handling and processing huge volumes of data can be challenging and at Report URI we can receive and process almost 65,000 JSON payloads per second, containing telemetry from clients all over the World.

I’m going to talk about our infrastructure and how we use the most awesome data structure that you’re probably not using, a Bloom Filter, to help us handle unimaginable amounts of data with amazing speed and space efficiency.