# AI Data Attacks

#### Introduction to AI Data

Artificial Intelligence (AI) systems are fundamentally data-driven. Consequently, their performance, reliability, and security inextricably linked to the quality, integrity, and confidentiality of the data they consume and produce.

#### The Data Pipeline: Fueling AI Systems

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1747626051992/9f740724-91de-43ac-aa2b-e6aca3ea758f.webp align="center")

At the heart of most AI implementations lies a `data pipeline`, a sequence of steps designed to collect, process, transform, and ultimately utilize data for tasks such as training models or generating predictions. While the specifics vary greatly depending on the application and organization, a general data pipeline often includes several core stages, frequently leveraging specific technologies and handling diverse data formats.

#### Data Collection

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1747626279992/679d3ba1-d3bf-440f-b53f-17dcb22d7fac.webp align="center")

The process begins with `data collection`, gathering raw information from various source. This might involve capturing user interactions from web applications as `JSON` logs streamed via messaging queues like `Apache Kafka`, ingesting structured transaction records from `SQL` databases like `PostgreSQL`, pulling sensor readings via MQTT from IoT devices, scraping public websites using tools like `Scrapy`, or receiving batch files (`CSV, Parquet`) from third parties. The collected data can range from images (`JPEG`) and audio (`WAV`) to complex semi-structured formats. The initial quality and integrity of this collected data profoundly impact all downstream processes.

#### Storage

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1747626649751/6a39bc92-60fb-4aba-bcd9-b6c09ef6782d.webp align="center")

Following collection, data requires `storage`. The choice of technology hinges on the data’s structure, volume, and access patterns. Structured data often resides in relational databases (`PostgreSQL`), while semi-structured logs might use `NoSQL` databases (`MongoDB`). For large, diverse datasets, organizations frequently employ `data lakes` built on distributed file systems (`Hadoop HDFS`) or cloud storage (`AWS S3, Azure Blob Storage`). Specialized databases like `InfluxDB` cater to time-series data. Importantly, trained models themselves become stored artifacts, often serialized into formats like Python’s `pickle` (`.pkl`), `ONNX`, or framework-specific files (`.pt, .pth, .safetensors`), each presenting unique security considerations if handled improperly.

#### Data Processing

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1747627078534/1467be7c-78ca-4466-bb36-c72f5d25142a.webp align="center")

Next, raw data undergoes `data processing and transformation`, as it’s rarely suitable for direct model use. This stage employs various libraries and frameworks for cleaning, normalization, and feature engineering. Data cleaning might involve handling missing values using `Pandas` and `scilit-learn`'s `Imputers`. Feature scaling often uses `StandardScaler` or `MinMaxScaler`. `Feature engineering` creates new relevant inputs, such as extracting date components or, for text data, performing tokenization and embedding generation using `NLTK` or `spaCy`. Image data might be augmented using `OpenCV` or `Pillow`. Large datasets often necessitate distributed processing frameworks like `Apache Spark` or `Dask`, with orchestration tools like `Apache Airflow` or `Kubeflow Pipelines` managing these complex workflows. The objective is to prepare a high-quality dataset optimized for the AI task.

#### Modeling

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1747627994708/a88480b0-86da-4bc3-b3e3-4e619513f104.webp align="center")

The processed data then fuels the `analysis and modeling` stage. Data scientists and ML engineers explore the data, often within interactive environments like `Jupyter Notebooks`, and train models using frameworks such as `scikit-learn`, `TensorFlow`, `Jax`, or `PyTorch`. This iterative process involves selecting algorithms (e.g., `RandomForestClassifier`, CNNs), tuning `hyperparameters`(perhaps using `Optuna`), and validating performance. Cloud platforms like `AWS SageMaker` or `Azure Machine Learning` often provide integrated environments for this lifecycle.

#### Deployment

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1747642511562/01aa8329-b1ec-4200-bc54-4ef099103525.webp align="center")

Once trained and validated, a model enters the `deployment` stage, where it’s integrated into a production environment to serve predictions. Common patterns include exposing the model as a `REST API` using frameworks like `Flask` or `FastAPI`, often containerized with `Docker` and orchestrated by `Kubernetes`. Alternatively, models might become serverless functions (`AWS Lambda`) or be embedded directly into applications or edge devices (using formats like `TensorFlow Lite`). Securing the deployed model file and its surrounding infrastructure is a key concern here.

#### Monitoring and Maintenance

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1747642800582/5d7d0d81-b42a-4e1f-956e-de35faf561ad.webp align="center")

Finally, `monitoring and maintenance` constitute an ongoing stage. Deployed models are continuously observed for operational health using tools like `Prometheus` and `Grafana`, while specialized ML monitoring platforms (`WhyLabs, Arize AI`) track `data drift, concept drift`, and prediction quality. Feedback from predictions and user interactions is logged and often processed alongside newly collected data to periodically `retrain` the model. This retraining is essential for adapting to changing patterns and maintaining performance but simultaneously creates a significant attack vector. Malicious data introduced via feedback loops or ongoing collection can be incorporated during retraining, enabling `online poisoning` attacks. Orchestration tools like `Airflow` often manage these retraining pipelines, making the security of data flowing into them critical.

#### Two Pipeline Examples

To clearly illustrate these complex pipelines, lets consider two examples:

First, an `e-commerce platform` building a `product recommendation system` collects user activity (`JSON` logs via `Kafka`) and reviews (`text`). This raw data lands in data lake (`AWS S3`). `Apache Spark` processes this data, reconstructing sessions and performing sentiment analysis (`NLTK`) on reviews, outputting `Parquet` files. Within `AWS SageMaker`, a recommendation model is trained on this processed data. The resulting model file (`pickle` format) is stored back in S3 before being deployed via a `Docker`\-ized `Flask` API on `Kubernetes`. Monitoring tracks click-through rates, and user feedback along with new interaction data feeds into periodic retraining cycles managed by `Airflow`, aiming to keep recommendations relevant but also opening the door for potential poisoning through manipulated feedback.

Second, a `healthcare provider` developing a `predictive diagnostic tool` collects anonymized patient images (`DICOM`) and notes (`XML`) from `PACS` and `EHR` systems. `Secure storage`(e.g., HIPAA-compliant `AWS S3`) is a requirement here. `Python` scripts using `Pydicom`, `OpenCV`, and `spaCy` process the data, standardizing images and extracting features. `PyTorch` trains a deep learning model (CNN) on specialized hardware. The validated model (`.pt` file) is securely stored and then deployed via an internal API to a clinical decision support system. Monitoring tracks diagnostic accuracy and data drift. While retraining might be less frequent and more rigorously controlled here, incorporating new data or corrected diagnoses still requires careful validation to prevent poisoning.

### AI Data Attacks

Having established the critical role of data and the structure of the data pipeline, we now focus specifically on `AI data attacks`. This module explores the techniques adversaries use to compromise AI systems by targeting the data itself; either during the training phase or by manipulating the stored model artifacts.

Unlike `evasion attacks` (`manipulating inputs to fool a deployed model`) or privacy `attacks` (`extracting sensitive information from a model`), the attacks covered here `fundamentally undermine the model’s integrity by its foundation`: the data it learn from or the format it’s stored in.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1747644484012/9cc5e250-355f-4fb1-aea1-985b118bb0fd.webp align="center")

Each stage of the data pipeline presents potential attack surfaces adversaries can exploit.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1747644545145/53725d59-d910-423c-80f2-3cda6cd13a70.webp align="center")

During `data collection`, the primary threat is initial `data poisoning`, where an attacker intentionally injects malicious data. This is a prime opportunity for introducing data intended for `label flipping` or `feature attacks`. In the e-commerce example, this could manifest as submitting fake positive reviews (poisoned features/labels) to boost a product’s recommendations or reviews with specific keywords (potential backdoor triggers) designed to cause unintended behavior later. For the healthcare scenario, an attacker might subtly alter `DICOM` metadata during ingestion or manipulate clinical notes, potentially mislabeling samples or embedding subtle feature perturbations. If this poisoned data infiltrates the training set, it can corrupt the resulting model according to the attacker’s goals.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1747644971659/cadd0a08-7cb5-40f2-b6cd-6120bfd9b908.webp align="center")

The `storage` stage faces traditional data security threats alongside model-specific risks, particularly relevant for `model stenography` and `Trojan` attacks. Unauthorized access to the `AWS S3` data lake or the healthcare provider’s secure storage could allow theft or tampering of training datasets, potentially modifying labels or features post-collection. Furthermore, stored model files (the `.pkl` recommendation model on `S3`, the `.pt` diagnostic model) are valuable targets. An attacker gaining write access could replace a legitimate model with a malicious one containing an embedded `trojan` or execute a `model stenography attack` by hiding code within the model file itself (leveraging insecure deserialization like `pickle.load()`), potentially compromising the `Flask` API server or the clinical system upon loading.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1747645427558/be0d5d64-dc9e-47da-bb53-1c5f8ccba4f9.webp align="center")

`Data processing` offers another avenue for manipulation, potentially facilitating `label flipping` or `feature attacks` even on initially clean data. If an attacker influences the cleaning, transformation, or feature engineering steps, they can corrupt data modeling. Compromising the e-commerce platform’s `Spark` job could lead to mislabeled review sentiments (`label flipping`), while manipulating the healthcare provider’s `Python` scripts could introduce subtle errors into standardized images or extracted test features (`feature attacks`), impacting the downstream model.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1747645743064/bc85032b-93a1-45d1-a208-741bd6585490.webp align="center")

The `analysis and modeling` stage is where the impact of `data poisoning` attacks introduced earlier becomes concrete. When the `AWS SageMaker` job trains the recommendation model on poisoned `Parquet` file containing flipped labels or perturbed features, or the `PyTorch` process trains the diagnostic CNN on data embedded with backdoor triggers, the resulting model learns the attack’s desired manipulations. It might learn incorrect patterns, exhibit biases, or contain hidden backdoors activated by specific inputs later.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1747646096800/5e03daea-1646-4376-8664-067ff11ac44c.webp align="center")

During `deployment`, the integrity of the model artifact crucial, concerning `Trojan` and `model stenography` risks. If the mechanism loading the model from storage (`S3`, secure file system) into the production environment is insecure, an attacker could inject a malicious model file at this point, achieving the same trojan effect or code execution via stenography as compromising the storage layer directly.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1747646334266/7cfff326-ff22-4168-8fc7-13414e501cdd.webp align="center")

Finally, the `monitoring and maintenance` stage, especially the common practice of `retraining` models, acts as a critical enabler for training data attacks like `online poisoning`. For example, the e-commerce platform’s `Airflow` retraining pipeline is a prime target. Attackers could continuously submit manipulated data: perhaps subtly altering clickstream data (`feature attacks`), submitting misleading feedback to influence future labels (`label flipping`), or injecting data designed to skew model weights towards particular outcomes over time. This gradual corruption degrades recommendation quality or introduces biases without needing initial dataset access.

The impact of successful AI data attacks can be severe, ranging from subtly biased decision-making and degraded system performance to complete model compromise and potentially enabling broader system breaches through embedded trojans.

#### Mapping Vulnerabilities to Security Frameworks

Leading security frameworks such as the `OWASP Top 10 for LLM Applications`, as highlighted in the Introduction to Red Teaming AI module, provides specific context for risks within the AI pipeline.

The major risk we are particularly focused on is `Data poisoning`, where attackers manipulate data during collection, processing, training, or feedback stages. This directly corresponds to `OWASP LLM03: Training Data Poisoning`.

Another relevant category of risk is the `AI Supply Chain`, addressed by `OWASP LLM05: Supply Chain Vulnerabilities`. This encompasses several related threats: compromising the integrity of bird-party data sources, tampering with pre-trained model artifacts (like injecting trojans), or exploiting vulnerabilities in the software components and platforms that make up the pipeline infrastructure itself. While `LLM05` covers many infrastructure aspects tied to components, robust protection also demands adherence to general secure system design principles beyond specific LLM list items, preventing unauthorized access throughout the pipeline. Ultimately, recognizing how both `Training Data Poisoning` and `Supply Chain Vulnerabilities` manifest is key to understanding the vulnerabilities in the AI data and model lifecycle.

Complementing OWASP’s specific vulnerability focus, `Google’s Secure AI Framework` (SAIF) provides a broader, lifecycle-oriented perspective. The data integrity issues identified map well onto SAIF’s core elements.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1748526704571/4195a04d-b251-48c8-8769-c713125f0dea.png align="center")

For instance, SAIF’s principles regarding `Secure Design`, securing `Data` components, and managing the `Secure Supply Chain` directly address the need to protect data throughout its lifecycle. Preventing `Data Poisoning` aligns with securing this `Data Supply Chain` and implementing rigorous `Security Testing` and validation during `Model` development, especially for data used in retraining. Likewise, maintaining model artifact integrity and preventing malicious code injection are central to SAIF’s `Secure Deployment` practices and verifying the `Secure Supply Chain`.

Finally, the challenge of monitoring for data or model manipulation, particularly within dynamic retraining loops, is covered by SAIF’s emphasis on `Secure Monitoring & Response`.