Design of Network Intrusion Detection Systems: An Effective Alarm Generation Perspective

Abstract

Intrusion Detection System, a hardware or software that monitors network or host activities for malicious behavior, is an indispensable component of system security. If an IDS deals with network activities (host activities) then it is called network based IDS (host based IDS). While signature and event based IDSs can detect known attacks only, anomaly based systems can detect both known and unknown attacks. An IDS is characterized by many parameters namely, effectiveness, efficiency, ease of use, security, transparency, interoperability etc. The thesis focusses at effectiveness, also called effective alarm generation, for all variants of network based IDSs namely, signature based, event based, header anomaly based and payload anomaly based systems. In signature based IDS, most of the alarms generated are false positives because signatures are generic and alarms are generated for all attack traffic which match some signature irrespective of the fact whether the attack could successfully exploit any vulnerability. To address this issue a false positive filtering scheme for signature based IDS has been proposed, which correlates alarms with network context information. As an enhancement to this filter, criticality of the application being targeted by an attack is examined, before eliminating the corresponding alarm estimated to be false positive. There is certain class of known attacks for which signatures cannot be written and so signature based IDSs cannot detect them. A novel event based IDS has been proposed for such attacks using the failure detection and diagnosis theory of discrete event systems. The working of the event based IDS has been illustrated on address resolution protocol based attacks. In header anomaly based IDSs, fairly high Detection Rate and Accuracy can be achieved for attacks detectable by header statistics. When the datasets to be processed by such systems are large, data summarization algorithms are applied. However, Detection Rate and Accuracy are not high in the systems that use data summarization algorithms compared to the ones which do not. A header anomaly based IDS for handling voluminous network traffic yet maintaining high Detection Rate and Accuracy has been proposed. Issues of effective alarm generation are more involved in payload anomaly based IDSs compared to the header based counterparts, which become more severe when training dataset has impurities. An impurity tolerant payload anomaly based IDS has been proposed using n-gram based statistical models. Tolerance is achieved by higher order n-grams and keeping their frequency information..