Is Weird really weird? Fatema Bannat Wala Security Engineer University

Is Weird really weird? Fatema Bannat Wala Security Engineer University of Delaware

About Me A big fan of Zeek IDS Have been working with Zeek for past 3.5 years Joined UD’s SecOps team in 2015 Passionate about Cyber-Security Also a part-time Ph.D. student

Roadmap of today’s talk What is Weird in Zeek? Motivation Types of Weirds in Zeek? Where to find Weirds? Deep dive into top most triggered Weirds Summary University’s Weird Profile

UD’s Traffic Profile 10 Gbps network links Average 5 Gbps network traffic per link Peak 8 Gbps per link 4 Zeek sensors, each getting 25% of total traffic Each sensor getting 300,000 pps Overall triggered Weird notices logged in weird.log were 21 M/day

What is Weird in Zeek? Google-land - “suggesting something supernatural; uncanny.” Zeek-land – “unusual or exceptional activity that can indicate malformed connections, traffic that doesn’t conform to a particular protocol, malfunctioning or misconfigured hardware, or even an attacker attempting to avoid/confuse a sensor.”

Motivation . Curiosity to know why so many ( 200) types of different Weirds getting triggered for the network traffic, whether they are serious events to be looked at, malformed packets or malformed application? Analyzing the Weird logs might reveal information about some activity that is hard to notice in the day to day life of network/security engineer.

Types of Weird There are MANY types of Weird defined in Zeek. More than 200 are triggered on our network traffic. Most common ones are: DNS RR unknown type Dns unmatched msg Dns unmatched reply fragment with DF bad ICMP checksum DNS Conn count too large possible split routing inappropriate FIN TCP Christmas truncated header data after reset data before established SYN seq jump SYN with data TCP ack underflow or misorder DNS truncated RR rdlength lt len line terminated with single CR DNS RR length mismatch

Where to find Weirds? Definition: Weird notices are mainly defined in following locations: In source code of Zeek IDS (in .cc files) In “script land”, in base/ policy/ folders (in various .bro scripts) Logging: The Weird notices are logged into a separate log file called “weird.log”. The logging of different Weirds is be controlled by base/frameworks/notice/weird.bro

Where to find Weirds? Definition ex: Core Layer- DNS.cc, Script Land – ssl/main.bro

Where to find Weirds? Logging : Script - weird.bro, Log file - weird.log base/frameworks/notice/weird.bro base/frameworks/notice/weird.bro ## ## AA table table specifying specifying default/recommended default/recommended actions actions per per weird weirdtype. type. const const actions: actions: table[string] table[string] of of Action Action {{ ["DNS RR bad length"] ACTION LOG, ["DNS RR bad length"] ACTION LOG, ["DNS RR length mismatch"] ACTION LOG, ["DNS RR length mismatch"] ACTION LOG, ["DNS RR unknown type"] ["DNS RR unknown type"] ACTION LOG, ACTION LOG, ["FIN after reset"] ACTION IGNORE, ["FIN after reset"] ACTION IGNORE, ["non IPv4 packet"] ACTION LOG ONCE, ["non IPv4 packet"] ACTION LOG ONCE, }} &default ACTION LOG &default ACTION LOG &redef; &redef;

Analysis of top most triggered Weirds In the University’s network the following were the top most triggered Weirds in weird.log over 24 hrs period: zcat weird.*.log.gz awk -F'\t' '{print 7}' sort uniq -c sort -rn 2,603,914 DNS RR unknown type 2,160,812 possible split routing 2,092,811 inappropriate FIN 753,398 fragment with DF 18,343 bad ICMP checksum

DNS RR unknown type Defined: In src/analyzer/protocol/dns/dns.cc Cause: The RR type currently not parsed in Bro Types seen in the traffic: Top most RR types recorded in weird.log 1,049,729 DNS RR unknown type 46 234,393 DNS RR unknown type 50 99,914 DNS RR unknown type 43 44,346 DNS RR unknown type 47 26,592 DNS RR unknown type 48

DNS RR unknown type After looking up those RR numbers, looks like they belong to DNSSEC protocol types:

DNS RR unknown type Reason: The DNS servers support DNSSEC and receive good amount of traffic for DNSSEC requests and responses. Remediation: Ignore those alerts and move on, or write DNSSEC RR types parsing support in Zeek*. We chose the later Result: Compiled the code with DNSSEC RR types parsing support and aha - no more alerts for those 5 RRs, the alert count for ‘DNS RR unknown type’ reduced from 2 M to 5 K per day. *The code with DNSSEC support in Zeek can be found at: https://github.com/zeek/zeek/tree/master or https://github.com/fatemabw/bro/tree/master (bro2.5.4 with DNSSEC)

possible split routing Defined: In src/analyzer/protocol/tcp/TCP.cc Cause: When Zeek doesn’t see the other side of the connection, signifying possible split routing Seen in the traffic: Top most destination hosts recorded in weird.log 1,515,526 128.xx.xx.xx 80 possible split routing 5,640 128.yy.yy.yy 443 possible split routing 121 5.185.16.24 443 possible split routing

possible split routing Reason: Router ACLs filtering one side of the connection, i.e. the requests from the internet to the server, so not to overwhelm the traffic from mirrored interface. Remediation: Router ACLs now filtering both sides of the connection to reduce noisy flows. “Don’t confuse your IDS” Result: Not seeing the possible split routing anymore for that server, just couple of those alerts every now and then for other IPs.

inappropriate FIN Defined: In src/analyzer/protocol/tcp/TCP.cc Cause: When Zeek sees a packet with a FIN set during a connection not complying with TCP/IP standard. Seen in the traffic: Top most destination hosts recorded in weird.log. 1,410,450 128.xx.xx.xx 80 inappropriate FIN 4,840 128.yy.yy.yy 443 inappropriate FIN 121 5.185.16.24 443 inappropriate FIN

inappropriate FIN Reason: Because of Router filtering one side of traffic, Zeek would see the Fin sent by the server as inappropriate, as it couldn’t see the other side of traffic (was a consequence of ‘possible split routing’). Remediation: Router ACLs now filtering both sides of the connection to reduce noisy flows. Result: Not seeing the inappropriate FIN anymore for that server, just couple of those alerts every now and then for other IPs

fragment with DF Defined: In src/Frag.cc Cause: When a fragmented packet is encountered with DF flag set (Don’t Fragment). Seen in the traffic: Top most Src hosts recorded in weird.log, the top most IPs belong to the DNS servers! 336,240 128.xxx.xxx.xx fragment with DF 335,411 128.yyy.yyy.yy fragment with DF 172 130.114.200.6 fragment with DF

fragment with DF Reason: Because of EDNS0 (Extensible DNS v0)! DNS approach to a reasonable compromise - the management of the packet MTU is passed into the application layer. The application will conventionally operate with a maximum UDP payload size (which is 4096B in BIND 9 by default) It was happening because of the lower MTU (1500) of the local interface of DNS servers. But as the servers support EDNS0 and DNSSEC, the responses are bound to be greater than local interface MTU.

fragment with DF Remediation: could try to increase the MTU of the local interfaces of DNS servers to be 4096 bytes, so that the fragmentation doesn’t happen on the local interface. could try to tweak maximum UDP payload size in the BIND settings, which is by default 4096, to 1472 bytes to match the MTU of the local interface to avoid fragmentation. (We chose this one)

fragment with DF Remediation continued BIND 9 configuration attribute: “max-udp-size” Sets the maximum EDNS UDP message size named will send in bytes. Valid values are 512 to 4096. The default value is 4096. This value applies to responses sent by a server. The usual reason for setting max-udp-size to a non-default value is to get UDP answers to pass through broken firewalls that block fragmented packets and/or block UDP packets that are greater than 512 bytes. Setting this to a low value will encourage additional TCP traffic to the nameserver.

fragment with DF Result: No more "fragment with DF" notices in the weird logs for the DNS servers! and the alert counts was reduced dramatically from 700K to couple of 100s for a day!!

Few other weirds . Following Weirds signify potentially bad traffic bad ICMP checksum (src/analyzer/protocol/icmp/ICMP.cc) TCP Christmas (src/analyzer/protocol/tcp/TCP.cc) Reason: bad ICMP checksum / TCP Christmas Weird notices were seen to be triggered by the scanner IPs, sweeping the range of IPs on the network. Remediation: Blocked the src IPs at the border. For bad ICMP checksum, we have a threshold of more than 10 bad checksum requests per host.

Summary of Weirds Weird Type DNS RR unknown type possible split routing inappropriate FIN fragment with DF bad ICMP checksum/TCP Christmas DNS unmatched msg/reply ( 11M/day) Improvement in Network Implemented DNSSEC RR parsing Fixed Router ACL filter Fixed Router ACL filter Fixed the BIND configuration Blocked the Bad IPs Not logging , doesn’t provide much insights *By above improvements we got rid of 18M/day Weird notices overall from weird.log file (85% reduction)

University’s Profile #Before count y Weird 2,603,914 753,398 164,344 13,480 0 DNS RR unknown type 1 fragment with DF 2 bad TCP checksum 3 DNS RR length mismatch 14,214 373 13,474 9,502 27,133 26,255 7,342 2 4 5 6 7 8 9 10 88 DNS truncated RR rdlength lt len premature connection reuse connection originator SYN ack data after reset empty http request truncated header Teredo bubble with payload HTTP content range unknown 2 1 1 1 89 90 91 92 smtp unmatched end of data bad IP checksum in tunnel bad TCP header len fragment size inconsistency Total plotted notices: 3,847,532

University’s Profile #After count 4,048 4,433 141,307 99,757 52,007 41743 2,584 24,120 25,876 11,749 52,388 1 1 2 1 6 y 0 1 2 3 4 5 6 7 8 9 10 93 94 95 96 97 Weird unknown SIP method inflate failed data after reset bad TCP checksum SYN seq jump DNS Conn count too large above hole data without any acks DNS RR length mismatch DNS truncated RR rdlength lt len data before established SYN with data unexpected multiple HTTP reqs UDP datagram length mismatch ssh unknown kex algorithm unknown protocol 7 TCP christmas Total plotted notices: 752,977

Acknowledgements Thanks to the EDUCAUSE Team for the support, and for the opportunity to be a presenter at SPC’19! Thanks everyone for your time and interest!

Session Evaluations There are two ways to access the session and presenter evaluations: 1 2 In the online agenda, click on the “Evaluate Session” link From the mobile app, click on the session you want from the schedule then click the associated resources and the evaluation will pop up in the list