How NetMap works¶

This page explains, step by step and with diagrams, what happens between "a packet crosses your network card" and "a dot appears on the map". No networking background is assumed. Short explanations of the terms are in the glossary at the end.

1. The big picture
2. The building blocks
3. Where packets come from
4. What happens to every packet
5. Finding real names for IP addresses
6. Putting hosts on the map
7. Colours: encrypted, unencrypted, alert
8. Security alerts
9. How the browser stays up to date
10. Live capture and the password prompt
11. Exports
12. Web API reference
Glossary

1. The big picture¶

Everything your computer does online (loading a website, a video call, an app checking for updates) is sent as small chunks of data called packets. Each packet carries the IP address of the sender and of the receiver. NetMap reads those packets, works out who each remote IP belongs to and where in the world it is, and draws it on a map.

flowchart LR
    subgraph IN["1 · Get packets"]
        L["Live capture<br/>(this computer, right now)"]
        F["Capture file<br/>(.pcap / .pcapng from Wireshark)"]
    end
    subgraph CORE["2 · Analyze (analyzer.py)"]
        A["Decode packet<br/>who → whom, which service"]
        N["Find a real name<br/>DNS · TLS SNI · HTTP · rDNS"]
        G["Look up location<br/>GeoIP database"]
        R["Check alert rules"]
    end
    subgraph OUT["3 · Show & share"]
        D["Live dashboard<br/>map · charts · tables"]
        E["Exports<br/>HTML · KML · CSV · TXT · JSON"]
    end
    L --> A
    F --> A
    A --> N --> G --> R
    R --> D
    R --> E

Everything runs on your computer. Captures are never uploaded. The only things fetched from the internet are map tiles and libraries for the web page, a one-time "what's my public IP" lookup, and optional reverse-DNS lookups. Both lookups can be switched off (--no-ip-lookup, --no-rdns).

2. The building blocks¶

NetMap is a small Python web application. The browser is just the screen: all analysis happens in Python.

flowchart TB
    subgraph LAUNCH["Launchers (one double-click)"]
        CMD["NetMap.command (macOS)"]
        BAT["start.bat (Windows)"]
        SH["run_dashboard.sh (Linux/macOS)"]
    end

    subgraph PY["Python backend"]
        APP["app.py<br/>Flask web server + Socket.IO<br/>sessions, uploads, exports"]
        AN["analyzer.py<br/>the engine: decoding, names,<br/>GeoIP, alerts, exporters"]
        HELP["capture_helper.py<br/>tiny sniffer that runs<br/>with admin rights"]
        CLI["main.py<br/>command-line version"]
    end

    subgraph DATA["Data files"]
        GEO[("GeoLiteCity.dat<br/>or *city*.mmdb")]
        PCAP[("sample-browsing.pcap<br/>demo-alerts.pcap")]
    end

    subgraph WEB["Browser (templates/ + static/)"]
        UI["index.html + dashboard.js + style.css<br/>Leaflet map · Chart.js chart"]
    end

    CMD --> SH
    SH -->|"creates .venv, installs deps,<br/>starts"| APP
    BAT -->|same| APP
    APP --> AN
    CLI --> AN
    AN --> GEO
    APP -->|"opens samples"| PCAP
    APP <-->|"raw packets over<br/>127.0.0.1 socket"| HELP
    APP <-->|"HTTP + WebSocket<br/>127.0.0.1:5050"| UI

Piece	Lines	Job
`analyzer.py`	~930	The engine. Knows nothing about the web, so the CLI and the server share it.
`app.py`	~600	Web server, live capture control, file uploads, pushing updates, exports.
`capture_helper.py`	~145	Reads packets with the system's libpcap as root and streams them to `app.py`. Uses only the standard library.
`main.py`	~85	Analyzes a file from the terminal and writes all reports.
`templates/index.html`, `static/js/dashboard.js`, `static/css/style.css`	~1300	The dashboard. The same files are embedded into the offline HTML report.

3. Where packets come from¶

There are three ways in, and all of them end in the same function: TrafficAnalyzer.ingest(timestamp, ip_packet, length).

flowchart LR
    W["Welcome screen"] --> LIVE["Watch live"]
    W --> OPEN["Open a capture<br/>(button or drag & drop)"]
    W --> SAMPLE["Try the sample"]

    LIVE -->|"already has capture rights<br/>(root, Windows+Npcap, readable /dev/bpf)"| SCAPY["Scapy sniff()<br/>inside app.py"]
    LIVE -->|"needs admin rights"| PROMPT["OS password prompt"] --> HELPER["capture_helper.py as root"]

    OPEN -->|"POST /api/pcap/upload<br/>saved to temp folder"| FILE["iter_capture_file()<br/>dpkt pcap / pcapng reader"]
    SAMPLE -->|"POST /api/pcap/open"| FILE

    SCAPY --> ING(["TrafficAnalyzer.ingest()"])
    HELPER --> ING
    FILE --> ING

Capture files are read twice. In the first, quick pass (infer_self_addresses), NetMap works out which public addresses belong to the computer that made the capture. With IPv6 there's no NAT, so your own address looks just like a server's. The trick is that the capturing machine talks to many peers, while each server usually talks only to it. The second pass does the real analysis. The progress bar shows 0–30 % for pass one and 30–100 % for pass two.

Supported link types: Ethernet, raw IP, BSD loopback, Linux "cooked" capture v1 and v2. 802.11 monitor-mode (radiotap) captures are rejected with a clear message.

4. What happens to every packet¶

flowchart TD
    P["Packet arrives"] --> IP{"Is it IPv4 / IPv6?"}
    IP -- no --> NONIP["count as non-IP<br/>(ARP etc.) and stop"]
    IP -- yes --> L4["Read transport: TCP / UDP / ICMP<br/>ports and TCP flags"]
    L4 --> FILT{"--protocol filter<br/>excludes it?"}
    FILT -- yes --> SKIP["count as filtered, stop"]
    FILT -- no --> TL["Add to per-second timeline<br/>(throughput chart)"]
    TL --> PAY["Look inside the payload for names<br/>DNS answer · TLS SNI · HTTP Host"]
    PAY --> SIDE{"Which side is 'me'?<br/>private IP, or one of my public IPs"}
    SIDE -- "both local" --> LAN["LAN-only traffic<br/>(counted, not mapped)"]
    SIDE -- "one remote" --> SVC["Name the service from the port<br/>443/TCP → HTTPS, 443/UDP → QUIC, 53 → DNS ..."]
    SVC --> EP["Update the remote host's record<br/>bytes sent/received, packets, ports, services,<br/>first/last seen, which of my devices"]
    EP --> NEW{"First time we see this host?"}
    NEW -- yes --> GEO["GeoIP lookup + queue reverse DNS"]
    NEW -- no --> FLOW
    GEO --> FLOW["Update the connection (flow)<br/>local, remote, protocol, port"]
    FLOW --> RULES["Run alert rules<br/>once per real connection"]

Two definitions decide what counts as "local":

Private / special addresses: 10.x, 172.16–31.x, 192.168.x, carrier-grade NAT 100.64.0.0/10, loopback, link-local and multicast (is_local()).
Your own public addresses: your public IP (auto-detected or --my-ip), the addresses of your network interfaces during live capture, and, for files, what the first pass inferred. IPv6 addresses are compared by their /64 prefix, because devices rotate addresses inside it.

5. Finding real names for IP addresses¶

An IP like 142.250.183.14 means little to most people. NetMap collects names from four sources and keeps the most trustworthy one:

flowchart LR
    subgraph S["Sources, most trusted first"]
        direction TB
        S3["③ TLS SNI<br/>the site name your browser<br/>sends when starting HTTPS"]
        S3b["③ HTTP Host header<br/>(plain HTTP only)"]
        S2["② DNS answer<br/>'youtube.com is at 142.250.x.x'"]
        S1["① Reverse DNS<br/>ask DNS 'what's the name<br/>for this IP?' (background)"]
    end
    S3 --> H["Hostname for the IP"]
    S3b --> H
    S2 --> H
    S1 --> H
    H --> O["Organisation guess<br/>'googlevideo' → YouTube,<br/>'fbcdn' → Meta, ..."]

A higher-priority name replaces a lower one, never the reverse (_set_hostname).
Even with HTTPS, the site name in the TLS ClientHello (SNI) is sent unencrypted. That's how NetMap names encrypted connections without decrypting anything. (Newer "Encrypted Client Hello" hides it, in which case DNS or reverse DNS fills in.)
Reverse DNS runs in a pool of 8 background threads, so it never slows down capture.
The organisation is guessed from about 60 hostname fragments (ORG_HINTS in analyzer.py).

6. Putting hosts on the map¶

flowchart LR
    IP["Remote IP"] --> PRIV{"Private / local?"}
    PRIV -- yes --> NONE["no location"]
    PRIV -- no --> DB{"Which database?"}
    DB -- ".mmdb found in folder<br/>(DB-IP / GeoLite2)" --> M["maxminddb<br/>IPv4 + IPv6"]
    DB -- "otherwise" --> L["GeoLiteCity.dat via pygeoip<br/>IPv4 only"]
    M --> LOC["city · region · country ·<br/>latitude · longitude"]
    L --> LOC
    LOC --> MAP["Map marker<br/>grouped by location,<br/>sized by data"]

"You" on the map is your public IP's location, or an exact point from --home lat,lon.
Results are cached per IP, so each address is only looked up once.
A GeoIP location is where an IP is registered or hosted. For big services that's usually a nearby CDN server, not the company headquarters. That's why YouTube can appear in Mumbai.
The bundled GeoLiteCity.dat has no IPv6 data, so IPv6 hosts are listed but not mapped. Dropping a free *city*.mmdb file in the folder fixes this (see the README).

7. Colours: encrypted, unencrypted, alert¶

Every remote host gets one status, which sets its colour everywhere: map, lists and KML.

flowchart TD
    H["Remote host"] --> A{"Any medium or high alert?"}
    A -- yes --> RED["🔴 Alert"]
    A -- no --> P{"Any plaintext service<br/>(HTTP, FTP, Telnet, POP3, IMAP, SMTP, LDAP)<br/>or a low alert?"}
    P -- yes --> YEL["🟡 Unencrypted"]
    P -- no --> E{"All app traffic encrypted?<br/>(HTTPS, QUIC, SSH, IMAPS, VPN ...)"}
    E -- yes --> GRN["🟢 Encrypted"]
    E -- no --> GRY["⚪ Other<br/>(DNS, NTP, unknown ports ...)"]

8. Security alerts¶

Rules run once per connection, after it's clearly real: an unanswered probe or a reset doesn't count, because scans have their own rule. Alerts are de-duplicated and capped at 500.

Rule	Severity	Triggers when	Why it matters
`port-scan`	High (inbound) / Medium (outbound)	One host sends TCP SYNs to 20+ different ports on another	Someone is looking for open services
`risky-port-4444`	High	Traffic on 4444	Default Metasploit/Meterpreter port
`risky-port-31337`	High	Traffic on 31337	Classic backdoor port
`risky-port-23`	High	Telnet to/from the internet	Passwords sent in clear text
`risky-port-445`	High	SMB to/from the internet	Worm and ransomware vector
`risky-port-*`	Medium / Low	1337, 6667 (IRC), 135, 139, 3389 (RDP), 5900 (VNC), 9001/9050 (Tor)	Commonly abused services
`plaintext`	Low	HTTP, FTP, POP3, IMAP, SMTP, LDAP	Anyone on the path can read the contents
`dga`	Low	A DNS name label is 20+ characters with high randomness (Shannon entropy > 4 bits) and no hyphens	Malware generates random domain names ("DGA") to find its server
`dns-tunnel`	Medium	DNS name over 100 characters, or a label over 50	Data being smuggled out inside DNS queries
`large-upload`	Medium	More than 100 MB sent to one host	Possible data exfiltration

Alerts are heuristics: hints of things worth a look, not proof of an attack. The synthetic demo-alerts.pcap triggers each rule type. It's generated by examples/make_demo_capture.py.

9. How the browser stays up to date¶

The browser never polls for data. The server pushes a fresh snapshot over a WebSocket (Socket.IO) whenever the data changes, at most once a second.

sequenceDiagram
    autonumber
    participant B as Browser (dashboard.js)
    participant S as app.py (Flask-SocketIO)
    participant A as TrafficAnalyzer
    B->>S: GET /  (page, CSS, JS)
    B->>S: WebSocket connect
    S-->>B: status + current snapshot
    loop every second (snapshot_pusher)
        S->>A: changed since last time? (version counter)
        alt data changed or live capture running
            A-->>S: snapshot(): summary, hosts, countries,<br/>services, timeline, alerts, DNS, feed
            S-->>B: 'snapshot' event
            B->>B: redraw map, chart, panels
        end
    end
    B->>S: POST /api/capture/start · /stop · /pcap/upload ...
    S-->>B: 'status' event (mode, progress, errors)

A snapshot is a plain JSON object with everything the UI needs. That's why the same dashboard.js can render the live server and the offline report: the report simply has a snapshot baked into the page as window.STATIC_SNAPSHOT.

10. Live capture and the password prompt¶

Reading raw network traffic needs administrator rights. Instead of asking users to run the whole app with sudo, NetMap runs as a normal user and starts only a tiny helper with admin rights, through the system's normal password dialog.

sequenceDiagram
    autonumber
    actor U as You
    participant B as Browser
    participant S as app.py (normal user)
    participant OS as macOS osascript /<br/>Linux pkexec
    participant H as capture_helper.py (root)
    participant N as Network card (libpcap)

    U->>B: click "Start live"
    B->>S: POST /api/capture/start
    S->>S: try to open capture device → permission denied
    S->>S: open a listening socket on 127.0.0.1:random<br/>and make a one-time random token
    S->>OS: run helper with admin rights (port, token, interface)
    OS->>U: system password prompt
    U->>OS: enter password
    OS->>H: start as root
    H->>S: connect, send token
    S->>S: token matches → accept
    H->>N: pcap_open_live(interface)
    H-->>S: 'L' frame: link type
    loop for every packet
        N-->>H: raw frame
        H-->>S: 'P' frame: timestamp, length, bytes
        S->>S: decode_link() → ingest()
    end
    U->>B: click "Stop"
    B->>S: POST /api/capture/stop
    S->>H: close the socket
    H->>H: notices the close, exits

Design details worth mentioning in a demo:

Least privilege: only ~145 lines of standard-library code run as root, never the web server.
macOS privacy protection blocks root processes from reading ~/Downloads, ~/Documents and ~/Desktop. So the helper's code is passed inline with python -c and run by the system Python, instead of being loaded from the project folder.
Token check: only the helper that NetMap started can feed it packets.
Wire format: each message is a 17-byte header type(1) · timestamp(8) · value(4) · length(4) plus payload. Types are L (link type), P (packet) and E (error).
Windows doesn't need a helper: with Npcap installed, Scapy captures directly.

11. Exports¶

All exporters take the same snapshot.

Export	Function	Open it with
Interactive report `.html`	`to_html_report`	Any browser, works offline. Same UI, data baked in. Good for sharing.
Google Maps / Earth `.kml`	`to_kml`	Google My Maps → Import, or Google Earth. Pins are grouped by country, with a line from you to each host.
Spreadsheet `.csv`	`to_csv`	Excel / Google Sheets. One row per host.
Text summary `.txt`	`to_text_report`	Any text editor.
Raw data `.json`	`to_json`	Your own scripts.

12. Web API reference¶

All endpoints are on http://127.0.0.1:5050 by default.

Method	Path	Does
GET	`/`	Dashboard page
GET	`/api/status`	Mode (idle/live/pcap), progress, errors, GeoIP info, capture access
GET	`/api/snapshot`	Current analysis as JSON
GET	`/api/interfaces`	Network interfaces (for the capture settings menu)
POST	`/api/capture/start`	Body `{"iface": "en0", "filter": "not port 22"}` (both optional)
POST	`/api/capture/stop`	Stop live capture
POST	`/api/reset`	Clear all data
GET	`/api/pcap/samples`	Capture files in the project folder
POST	`/api/pcap/open`	Body `{"name": "sample-browsing.pcap"}`. Only names from the list above are accepted.
POST	`/api/pcap/upload`	Multipart field `file` (.pcap/.pcapng/.cap, up to 2 GB)
GET	`/api/export/<fmt>`	`html`, `kml`, `csv`, `txt` or `json`
WebSocket	Socket.IO events `status`, `snapshot`	Pushed by the server

Glossary¶

Term	Meaning
Packet	A small chunk of data sent over a network. A web page is hundreds of them.
IP address	The "postal address" of a device on the internet, e.g. `142.250.183.14` (IPv4) or `2404:6800:...` (IPv6).
Private IP	An address used only inside your home or office network (`192.168.x.x`, `10.x.x.x`). Your router translates it to one public IP (NAT).
Port	A number that says which service a packet is for: 443 = HTTPS, 53 = DNS, 22 = SSH ...
TCP / UDP	The two main transport protocols. TCP is reliable and ordered. UDP is fast and fire-and-forget, and is used by DNS and QUIC.
QUIC	Modern encrypted protocol over UDP 443, used by Google, YouTube and others (HTTP/3).
DNS	The internet's phone book: turns `youtube.com` into an IP address.
Reverse DNS	Asking DNS the opposite question: "what's the name for this IP?"
TLS / SNI	TLS is the encryption behind HTTPS. SNI is the site name the browser sends in the clear when starting TLS.
PCAP / PCAPNG	File formats for saved packet captures (what Wireshark saves).
libpcap / Npcap	The system libraries that actually read packets off the network card (Npcap on Windows).
BPF filter	A capture filter expression, e.g. `not port 22` or `host 8.8.8.8`.
GeoIP	A database that maps IP ranges to approximate locations.
CDN	Content Delivery Network: copies of a service placed in many cities so it loads fast near you.
Flow	One conversation: (your device, remote host, protocol, port).
Shannon entropy	A measure of randomness. `google` is low, `xj4k9qz7v2mplw8rt5nb3c` is high.