基于数据分析的网络安全（影印版）

《基于数据分析的网络安全（影印版）》分成3个部分，包括采集和组织数据的流程、多种分析工具以及几个不同的分析场景和技术。它很适合网络管理员和熟悉脚本的运行安全分析员。传统的入侵检测和日志分析已经不足以保护今天的复杂网络。在这本实用指南里，安全研究员Michael Collins为你展示了多个采集和分析网络流量数据集的技术和工具。你将理解你的网络是如何被利用的以及有哪些必要手段来保护和改善它。

Preface
PartI.Data
1.Sensors and Detectors： An Introduction
Vantages： How Sensor Placement Affects Data Collection
Domains： Determining Data That Can Be Collected
Actions： What a Sensor Does with Data
Conclusion
2.Network Sensors
Network Layering and Its Impact on Instrumentation
Network Layers and Vantage
Network Layers and Addressing
Packet Data
Packet and Frame Formats
Rolling Buffers
Limiting the Data Captured from Each Packet
Filtering SpeciFic Types of Packets
What Iflt's Not Ethernet？
NetFlow
NetFlow v5 Formats and Fields
NetFlow Generation and Collection
Further Reading
3.Host and Service Sensors： Logging Traffic at the Source
Accessing and Manipulating LogFiles
The Contents of Logfiles
The Characteristics of a Good Log Message
Existing Logflles and How to Manipulate Them
Representative Logflle Formats
HTTP： CLF and ELF
SMTP
Microsoft Exchange： Message Tracking Logs
Logfile Transport： Transfers，Syslog，and Message Queues
Transfer and Logfrle Rotation
Syslog
Further Reading
4.Data Storage for Analysis： Relational Databases，Big Data，and Other Options
Log Data and the CRUD Paradigm
Creating a Well—Organized Flat File System： Lessons from SiLK
A Brieflntroduction to NoSQL Systems
What Storage Approach to Use
Storage Hierarchy，Query Times，and Aging
Partll.Tools
5.The SiLK Suite
What Is SiLK and How Does It Work？
Acquiring and Installing SiLK
The DataFiles
Choosing and Formatting Output Field Manipulation： rwcut
Basic Field Manipulation： rwfrlter
Ports and Protocols
Size
IP Addresses
Time
TCP Options
Helper Options
Miscellaneous Filtering Options and Some Hacks
rwfileinfo and Provenance
Combining Information Flows： rwcount
rwset and IP Sets
rwuniq
rwbag
Advanced SiLK Faalities
pmaps
Collecting SiLK Data
YAF
rwptoflow
rwtuc
Further Reading
6.An Introduction to R for Security Analysts
Installation and Setup
Basics of the Language
The R Prompt
R Variables
Writing Functions
Conditionals and Iteration
Using the R Workspace
Data Frames
Visualization
Visualization Commands
Parameters to Visualization
Annotating a Visualization
ExportingVisualization
Analysis： Statistical Hypothesis Testing
Hypothesis Testing
Testing Data
Further Reading
7.Classification and Event Tools： IDS，AV，and SEM
How an IDS Works
Basic Vocabulary
Classifler Failure Rates： Understanding the Base—Rate Fallacy
Applying ClassiFication
Improving IDS Performance
Enhancing IDS Detection
Enhanang IDS Response
Prefetching Data
Further Reading
8.Reference and Lookup： Tools for Figuring Out Who Someone ls
MAC and Hardware Addresses
IP Addressing
IPv4 Addresses，Theu Structure，and Significant Addresses
IPv6 Addresses，Their Structure and Significant Addresses
Checking Connectivity： Using ping to Connect to an Address
Tracerouting
IP Intelligence： Geolocation and Demographics
DNS
DNS Name Structure
Forward DNS Querying Using dig
The DNS Reverse Lookup
Using whois to Find Ownership
Additional Reference Tools
DNSBLs
9，More Tools
Visualization
Graphviz
Communications and Probing
netcat
nmap
Scapy
Packet Inspection and Reference
Wireshark
GeoIP
The NVD，Malware Sites，and the C*Es
Search Engines，Mailing Lists，and People
Further Reading
Partlll.Analytics
10.Exploratory Data Analysis and Visualization
The Goal of EDA： Applying Analysis
EDA Workflow
Variables and Visualization
Univariate Visualization： Histograms，QQ Plots，Boxplots，and Rank Plots
Histograms
Bar Plots（Not Pie Charts）
The Quantile—Quantile（QQ）Plot
The Five—Number Summary and the Boxplot
Generating a Boxplot
Bivariate Description
Scatterplots
Contingency Tables
Multivariate Visualization
Operationalizing Security Visualization
Further Reading
11.On Fumbling
Attack Models
Fumbling： Misconfiguration，Automation，and Scanning
Lookup Failures
Automation
Scanning
Identifying Fumbling
TCP Fumbling： The State Machine
ICMP Messages and Fumbling
Identifying UDP Fumbling
Fumbling at the Service Level
HTTP Fumbling
SMTP Fumbling
Analyzing Fumbling
Building Fumbling Alarms
Forensic Analysis of Fumbling
Engineering a Network to Take Advantage of Fumbling
Further Reading
12.Volume and Time Analysis
The Workday and Its Impact on Network Traffic Volume
Beaconing
File Transfers／Raiding
Locality
DDoS，Flash Crowds，and Resource Exhaustion
DDoS and Routing Infrastructure
Applying Volume and Locality Analysis
Data Selection
Using Volume as an Alarm
Using Beaconing as an Alarm
Using Locality as an Alarm
Engineering Solutions
Further Reading
13.Graph Analysis
Graph Attributes： What Is a Graph？
Labeling，Weight，and Paths
Components and Connectivity
Clustering Coeffiaent
Analyzing Graphs
Using Component Analysis as an Alarm
Using Centrality Analysis for Forensics
Using Breadth—First Searches Forensically
Using Centrality Analysis for Engineering
Further Reading
14.Application Identification
Mechanisms for Application Identification
Port Number
Application Identiflcation by Banner Grabbing
Application Identification by Behavior
Application Identification by Subsidiary Site
Application Banners： Identifying and Classifying
Non—Web Banners
Web Client Banners： The User—Agent String
Further Reading
15.Network Mapping
Creating an Initial Network Inventory and Map
Creating an Inventory： Data，Coverage，and Files
Phase Ⅰ： The First Three Questions
Phase Ⅱ： Examining the IP Space
Phase Ⅲ： Identifying Blind and Confusing Traffic
Phase Ⅳ： Identifying Clients and Servers
Identifying Sensing and Blocking Infrastructure
Updating the Inventory： Toward Continuous Audit
Further Reading
Index

CHAPTER 2 Network Sensors
A network sensor collects data directly from network traffic without the agency of anintermediary application，making them different from the host—based sensors discussedin Chapter 3．Examples include NetFlow sensors on a router and sensors that collecttraffic using a sniffing tool such as tcpdump．
The challenge ofnetwork traffic is the challenge you face with all log data：actual securityevents are rare，and data costs time and storage space．Where available，log data ispreferable because it‘s clean（a high—level event is recorded in the log data）and compact．The same event in network trafflC would have to be extracted from millions of packets．which can often be redundant，encrypted，or unreadable．At the same time，it is veryeasy for an attacker to manipulate network traffic and produce legitimate—looking butcompletely bogus sessions on the wire．An event summed up in a 300一byte log recordcould easily be megabytes of packet data，wherein only the first 1 0 packets have anyanalytic value．
That’s the bad news．The good news is that network traffiC‘S“protocol agnosticism；’forlack of a better term，means that it is also your best source for identifying blind spots inyour auditing．Host—based collection systems require knowing that the host exists in thefirst place，and there are numerous cases where you’re likely not to know that a particularservice is running until you see its traffic on the wire．Network traffic provides a viewof the network with minimal assumptions--it tells you about hosts on the network youdOn‘t know existed、backdoors you weren’t aware of，attackers already inside your bor—der，and routes through your network you never considered．At the same time，whenvou face a zero—day vulnerability or new malware，packet data may be the only datasource you have．
……