Why is this existing?


Spidey Sense is a tool to help security practitioners to discover threats and malicious activity in HTTP logs. It ingests the raw logs from an HTTP server, augments them with the data from threat intelligence databases and finds patterns that look suspicious. These findings help fight abuse and improve security posture.

Ingest


The data can be ingested using a simple HTTP API right from the web server. It's a POST request to the /v1/ingest endpoint with the logs uploaded as a file.

Authentication and authorization is performed through HTTP Basic Auth. Every account receives a unique API key that can be viewed through the interface. The user name is key and the password is the API key itself. Here is how you use basic auth with curl:

curl -s --user 'key:API-KEY' \
    https://api.spidey-sense.com/v1/ingest -F logs=@access.log

Log Format


The service accepts the logs in Common Log Format and should work out of the box for Apache and Nginx access logs.

Here is how log record in Common Log Format looks:

172.18.0.1 - - [29/Feb/2020:21:46:57 +0000] "GET /docs HTTP/1.1" 200 10564 "-" "Mozilla..." in 11357µs
  • 172.18.0.1 is the IP address of the client (remote host) which made the request.
  • [29/Feb/2020:21:46:57 +0000] is the date, time, and timezone that the request was received.
  • "GET /docs HTTP/1.1" is the request for the client defined by method, resource and protocol.
  • 200 is the HTTP status code returned by the server.
  • 10564 is the size of the object returned to the client, measured in bytes.
  • "Mozilla..." is the User-Agent header provided by the client. Usually it's a long string, we clipped it here to save space.
  • 11357µs is the time it took for the server to process the request.

We are working on the algorithm that would automatically detect the logs structure. Meanwhile please feel free to contact us if you want to request a specific format.

Visitors


Every log record represents a Hit. It's a single request to the server.

Using special algorithm Spidey Sense combines Hits into Visitors and categorizes them. A Visitor describes a single user accessing the server.

Each visitor is assigned a kind that can be one of the following:

  • - Browser, a known Web Browser. Example: Mozilla Firefox on Linux.
  • - Bot, a known bot or crawler. Example: Google-Bot.
  • - Script, a script accessing the server remotely. It often times a signal of an active exploit.

The information about the visitors gets augmented with threat intelligence to help with categorization. To filter out suspicious activity the visitors are assigned one of the following categories:

  • - Benign, a legitimate visitor of the service.
  • - Suspicious, a visitor that exhibits a weird behaviour and needs to be scrutinized.
  • - Malicious, a visitor that is trying the exploit the service maliciously.

Future updates


We are constantly working on improving the service, so please don't hesitate to contact us with any feedback.