Methodology and Technicalities

Data is collected on my server which is managed by caddy right now. Previously, it has been nginx but caddy just seemed more intuitive plus without a need to manage certificates.

Data Processing

My technology stack for this project:

• I've written a small tool called glog using Go. Repo is here. glog processes, parses and exports my logs into a DuckDB database
• I use DuckDB for data exploration and transformation plus preparing it for data visualization.
• Visualization/Presentation part is done with Observable Framework

Enrichment and Transformations

Fingerprints

At the moment fingerprints for request headers and UA are simply md5 hashes done in DuckDB. There are more sophisticated methods, I plan to revisit this topic later.

URI Classification

Simplistic approach is to classify as you go, most seen patterns are added and new patterns will be added in the future.