Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement FQDN Beaconing using TLS SNI and HTTP Host #739

Merged
merged 25 commits into from
Aug 1, 2022

Conversation

Zalgo2462
Copy link
Contributor

@Zalgo2462 Zalgo2462 commented Jul 1, 2022

This PR implements an FQDN beaconing analysis based off of TLS SNI and HTTP Host headers.
The following collections are added:

  • SNIconn: equivalent of the uconn collection for src -> fqdn pairs
    • contains dat array subdocuments dat.tls and dat.http
    • dat.tls entries record TLS connection data for the current chunk
    • dat.http entries record HTTP connection data for the current chunk
  • beaconSNI: equivalent of the beacon collection for src -> fqdn pairs
    • For each document in SNIConn, the connection data for TLS and HTTP is merged to create a connection dataset for FQDN beaconing analysis
    • Same schema as beacon collection except for the destination is an fqdn entry instead of a destination unique IP

Additions:

  • TLS SNI and HTTP connection parsing logic
    • Three new maps in the ParseResults: TLSConnMap, HTTPConnMap, and ZeekUIDMap.
    • ZeekUIDMap is used to link in information from the conn.log file into the TLSConnMap and HTTPConnMap entries
      • If UIDs are not present for some reason (alternative data sources), the program should not crash
  • The sniconn package creates uconn like entries for every source -> fqdn pair in the TLSConnMap and HTTPConnMap
  • The beaconSNI package creates beacon like entries for every source -> fqdn pair in the TLSConnMap and HTTPConnMap which passes the connection threshold
  • The beaconSNI package creates dat.mbsni and dat.max_beacon_sni entries in the host collection with a separate summary step
  • The show-beacons-sni command writes the results of the beaconSNI analysis out to the terminal
  • The rita-html-report now writes out a separate tab for Beacon SNI analysis
  • The configuration file now has a new section for controlling the SNI beaconing analysis

Removals:
- I have left the old FQDN analysis code in the project during testing of the PR for comparison sake
- I will remove it after we approve the new code

Bug fixes:

  • The proxy beacons package was consulting the wrong beacon connection threshold in the config. This has been fixed.

Oddities:

  • The strobe implementation is a little different than for uconns. Instead of a single top level strobe field, strobe markers are stored in dat.http.strobe, dat.tls.strobe, and dat.beacon.strobe.

Todo:

  • sniconn Readme
  • beaconsni Readme

@caffeinatedpixel caffeinatedpixel self-requested a review July 28, 2022 19:28
@caffeinatedpixel
Copy link
Contributor

Is the entire beacon fqdn package being removed or only a portion of it?

@Zalgo2462
Copy link
Contributor Author

We're going to keep the old fqdn beaconing in for this release.

Copy link
Contributor

@caffeinatedpixel caffeinatedpixel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Went through the code and noticed some oddities. Still working on testing.

pkg/beaconsni/Readme.md Outdated Show resolved Hide resolved
pkg/sniconn/Readme.md Outdated Show resolved Hide resolved
pkg/beaconsni/analyzer.go Outdated Show resolved Hide resolved
pkg/sniconn/Readme.md Outdated Show resolved Hide resolved
pkg/sniconn/Readme.md Outdated Show resolved Hide resolved
pkg/beaconsni/Readme.md Outdated Show resolved Hide resolved
pkg/beaconsni/dissector.go Show resolved Hide resolved
{Key: []string{"src", "fqdn", "src_network_uuid"}, Unique: true},
{Key: []string{"src", "src_network_uuid"}},
{Key: []string{"fqdn"}},
{Key: []string{"responding_ips.ip", "responding_ips.network_uuid"}},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do not create indexes for resolved_ips in beaconFQDN. Do you remember any contradiction for adding them in beaconFQDN or was this just an oversight? I have a faint memory of the index causing a slowdown instead of a speed increase, but I seriously could have been dreaming.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll go take a brief skim through our past tickets, but this doesn't ring a bell for me. I don't think I was heavily involved with the development of beaconFQDN, so I might not be the best person to ask though.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't find anything related to it while sifting through old issues & PRs, so I'll just blame it on the mandela effect.

pkg/beaconsni/results.go Outdated Show resolved Hide resolved
pkg/sniconn/mongodb.go Outdated Show resolved Hide resolved
Copy link
Contributor

@caffeinatedpixel caffeinatedpixel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requested changes were made. Stated functionality works as expected. SNI beaconing command line commands & html report functionality work as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants