BSc Dissertation · AI / Machine Learning

Cybersecurity Incident Classification — NLP, ML & MITRE ATT&CK

Research and evaluation of an AI-assisted cybersecurity incident-classification pipeline using text features, transformer embeddings, structured security metadata, and ensemble machine-learning models.

BSc dissertation / research project

Context & Scope

This project was developed as part of my BSc (Hons) Cyber Security dissertation. It investigates how NLP and machine-learning techniques can support security incident classification and triage. It is a research prototype and is not presented as a production SOC automation platform.

Objective

Develop and evaluate a reproducible incident-classification pipeline that combines textual and structured security features, compare multiple machine-learning models, and assess their potential relevance to security triage workflows.

Research Approach

Prepared a working research subset of 50,000 cybersecurity incident records from the Microsoft GUIDE dataset.
Combined TF-IDF text representations, Sentence Transformer embeddings, and structured security metadata into a hybrid feature pipeline.
Used SMOTE as part of the experimental workflow to address class imbalance.
Benchmarked Random Forest, Gradient Boosting, and XGBoost classifiers.
Evaluated model behaviour using accuracy, weighted F1-score, cross-validation, confusion matrices, ROC analysis, precision-recall analysis, and feature importance.
Used MITRE ATT&CK context to support security-oriented interpretation of incident categories and model output.

Key Results

The experimental dataset contained 50,000 security incident records.
The engineered hybrid representation contained 694 features.
Three ensemble machine-learning classifiers were evaluated.
Random Forest achieved the strongest reported test accuracy at 69.28%.
Random Forest achieved a weighted F1-score of 0.6963.
Reported cross-validation accuracy for the strongest model was 75.96%.

Research Evidence

Selected figures from the original BSc dissertation experiment. These results represent research evaluation, not production SOC performance.

Experimental cybersecurity incident classification machine-learning framework — Experimental pipeline combining GUIDE data, NLP feature engineering, structured security metadata, SMOTE, and three ensemble classifiers.

Accuracy comparison of Random Forest Gradient Boosting and XGBoost — Reported test-accuracy comparison across the three evaluated classifiers, with Random Forest producing the strongest result.

Random Forest cybersecurity incident classification confusion matrix — Random Forest confusion matrix used to inspect class-level performance, imbalance, and misclassification behaviour.

Research Boundary

The results come from an academic research pipeline evaluated on a prepared dataset. They do not demonstrate production deployment, autonomous SOC decision-making, or operational performance against live enterprise incidents.

Security Relevance

The project demonstrates security-data preparation, NLP feature engineering, transformer-based representations, class-imbalance handling, model comparison, performance evaluation, MITRE ATT&CK-aware analysis, and critical assessment of AI-assisted security triage.