small 'best of' download files instead of full 10 gb...
in one of the recent blogs from you or cloudflare, it is talked that basically it would be best to deny all passwords with a count > 100 and warn on password > 20. would it be possible to create download files just for these (i think) like 10 mil records (all > 20)? that would make it easier to create a local repository database with a workable download size and working count. ... and ignoring the rare passwords which make up the largest bucket of your collection.
Lukas
shared this idea
You can easily do this yourself by pulling down the entire data set then just extracting all records within the threshold you’ve chosen. I don’t want to publish multiple versions of the same data at different thresholds, this is a very subjective decision and it can easily be extracted from the existing data,
-
Get a VPS for a couple of bucks, filter out the data, job done! If you don't have the means to do that then I suggest you're probably not running a service that will really benefit from defences against credential stuffing.
-
Lukas
commented
ok, i understand. but it is just that not everybody has the means to download and open a 30 gb textfile and process files that large. but ok... :) thanks for the answer. and many thanks for the generall effort in the project in general!