Implementing fail2ban style monitoring on Node.js gateway and accidentally banning the Google bot(!) скачать в хорошем качестве

Implementing fail2ban style monitoring on Node.js gateway and accidentally banning the Google bot(!)

In this video I show some results of a fail2ban style monitoring implementation I wrote on top of a Node.js gateway. What I don't discuss in the video is the importance of structuring logs in such as a way as to 1) keep the entire log on one line 2) tag each request / response with the IP, timestamp, request ID, session ID 3) usage of WAF or OWASP as options to creating dictionaries of reference 4) log aggregation into a separate DB for separate analysis. I also don't discuss the techniques that went into crafting this fail2ban style monitoring. The most important thing to mention is that it's very important to be mindful that on a gateway, you want to be very sensitive to CPU time per request. Therefore, what you see in the video is that I use a technique that 1) stores timestamps of occurrence into two arrays 2) monitor a count 3) once count exceeds threshold, e.g. every 10 requests 4) if error count exceeds threshold as well, e.g. at least 30 errors 5) check if the n-30 item in the array (a timestamp) is within threshold. This approach uses a minimal enough memory footprint but more importantly optimizes to be cheap on computation compared to other obvious approaches; such as using timeouts or cleaning the error count buffer or cache on each new error; doing any kind of array traversal on a ban check for counts within the specified time interval. However, this shows a simple way to quickly block bots that quickly hit your server with a flurry of exploitation techniques.