AI and data protection: Using AI systems in compliance with the GDPR

What is artificial intelligence (AI)?

It is already being used by companies in a variety of ways to increase efficiency, save time and reduce costs. AI systems are used in customer service and content production (e.g. chatbots such as ChatGPT), accounting, and HR (e.g. the application process).

AI has become an integral part of everyday corporate life. But what does the term actually mean? The AI Act, which came into force on 1 August 2024, provides the first legal definition of AI in Article 3. 3 para. 1. According to this definition, an AI system must exhibit the following seven characteristics:

1. Machine-supported system
2. Autonomous operation (to varying degrees)
3. Potential adaptability
4. Explicit or implicit goals
5. The ability to derive (inference)
6. Outputs
7. Influencing the physical or virtual environment

In summary, AI systems are autonomous and adaptive with the ability to draw conclusions. This definition is very broad and the subject of much controversy. According to recital 12, simple conventional software should not be included; however, this cannot be clearly derived from the official definition of the AI Act. The Commission specified the definition in the non-binding "Commission Guidelines on the definition of an artificial intelligence system" to clarify the concrete delimitation of AI technology.

What role does data protection play in the use of AI?

The performance of AI systems largely depends on the quality of the underlying data. In addition to being high quality, a sufficient quantity of data is also required to achieve reliable results. This data often contains personal information that is protected by the strict provisions of the General Data Protection Regulation (GDPR). Therefore, personal data can be processed at various points when using AI systems. This makes data protection a central aspect of AI usage.

Any processing of personal data — whether during the collection of training data, AI training, or productive operation involving input and output — must be legally justified under data protection law (e.g. consent, contract, legitimate interests).  

In addition to the principle of lawfulness set out in Art. 5 GDPR, other data protection principles are also highly relevant. These include the transparency requirement, purpose limitation, data accuracy requirements, and data minimisation.

Which laws are relevant when using AI?

The use of AI can fall within the scope of various laws. In addition to the AI Act, the focus is on data protection and copyright law, for example.

The AI Act applies if AI systems or general-purpose AI models (GPAIs), such as large language models (LLMs), are offered in the EU or if AI systems are put into operation or used. The AI Act applies alongside other relevant laws and does not supersede them.

In the area of data protection, the GDPR and the BDSG apply if personal data is processed for AI training or during the operational use of AI. The Copyright Act also applies when copyrighted works are used for AI training, or when the AI system is monetised as a copyrighted work.

In the healthcare sector, the Medical Device Regulation (MDR) and the European Health Data Space (EHDS) must also be observed. AI systems can be covered by the MDR as software for medical purposes, which would make the AI Act applicable.

If AI is used for advertising purposes, the UWG must also be observed.

The Commission has decided not to pursue the planned AI Liability Directive, which would have expanded national liability law to include AI-specific tort law.

What is the AI Act?

The EU has created the world's first comprehensive legal framework for artificial intelligence (AI) with the AI Act. It is a technology-specific and product safety regulation for AI. It was published in the Official Journal of the EU on 12 July 2024 and came into force on 1 August 2024. The application of the AI Act is subject to staggered transition periods, with the first regulations coming into effect on 2 February 2025.

The AI Act aims to strengthen confidence in AI, promote innovation and ensure the safety and fundamental rights of EU citizens are safeguarded when using AI.

The AI Act takes a risk-based approach to prevent excessive regulation. To this end, it imposes compliance obligations on AI systems, with greater obligations applying to systems with a greater potential to interfere with fundamental rights. High-risk AI systems are at the heart of the AI Act.

The obligations that apply to companies using AI tools depend on whether they are providers or operators under the AI Act. Providers are defined as those who develop an AI system or have it developed, and then place it on the market or put it into operation. The same applies if they develop General Purpose AI (GPAI) or have it developed and place it on the market.

Operators are individuals who use an AI system under their own responsibility. Providers of AI systems must fulfil a significantly more extensive catalogue of obligations than operators, so it is essential to define the company's role precisely. Under the special conditions of Art. 25 of the AI Act, operators may also assume a provider role.

How does the GDPR apply to AI?

Legal basis and purpose limitation

The EU legislator has expressly clarified in Recital 63 of the AI Act that the AI Act itself does not constitute a legal basis for processing personal data unless the AI Act expressly provides otherwise. The processing of personal data must therefore be based on Art. 6 or Art. 9 of the GDPR.

One authorised form of data processing is the prior informed consent of the data subject in accordance with Art. 6 para. 1 lit. a GDPR. Companies that wish to process personal data for AI training purposes must obtain relevant consent from data subjects. A challenge here is that consent can be withdrawn at any time by the data subjects, meaning the personal data must be deleted by the controller.

However, if personal data remains in the neural network after processing, it is generally not technically feasible to erase it selectively. Obtaining consent can also present challenges for companies. For example, if data is collected using web scraping, or if the AI accesses the internet during operation, it is practically impossible to obtain consent due to a lack of contact options.

In practice, many companies therefore rely on the legal basis of legitimate interest in accordance with Art. 6 para. 1 lit. f GDPR. This states that processing is permitted if it is necessary to safeguard the legitimate interests of the controller or a third party, provided that the interests or fundamental rights and freedoms of the data subject are not outweighed by this.

The concept of legitimate interest is broadly interpreted, taking into account increases in efficiency or cost savings, as well as the optimisation of own offers through AI systems, including their further development. However, the legal basis of legitimate interest has its limitations when it comes to processing the data of minors or sensitive data, such as health data.

If data already available in the company is to be used for the training of AI, the principle of purpose limitation must be observed. This means that the purposes of data processing must already be defined, clear and legitimate when personal data is collected. Most of the ‘inventory data’ available in companies was not collected for the purpose of AI training, meaning that personal data may not be used for AI training without further ado, as this would constitute a change of purpose.

A change of purpose is only permitted if the new processing purpose is compatible with the original one. Among other things, the context in which the personal data was collected, the type of personal data or the possible consequences of the intended further processing for the data subjects must be taken into account. The data subjects must also be informed about the change of purpose.

Data minimisation & transparency

Large amounts of data (big data) are required for AI training. However, the principle of data minimisation states that personal data must be adequate, relevant and limited to what is necessary for the purposes of the processing. However, the principle of data minimisation does not aim to keep the amount of personal data to be processed as low as possible.

Instead, the extent to which fundamental rights are interfered with should be limited to the data that is actually required for the intended purpose, unless there is another reasonable way of achieving that purpose with less interference. To resolve this tension, data without personal references, such as synthesised or anonymised data, should be used primarily for AI training.

Alternatively, data protection-friendly training methods, such as federated learning, can be employed. Overall, depending on the application's purpose, the AI model must only contain and reproduce the minimum amount of personal data required for training.

The transparency principle states that personal data must be processed 'in a way that is comprehensible to the data subject'. This principle is intended to enable data subjects to exercise their rights. This principle is reflected in the information obligations and rights of access in Articles 13–15 GDPR. Data subjects can only make informed decisions about the processing of their personal data if they have sufficient information.

At first glance, this does not appear to be compatible with highly complex artificial intelligence models. This is because deep learning systems become increasingly opaque, or no longer fully comprehensible, as the learning process progresses. This is referred to as a 'black box'. This lack of transparency means that disclosure and information obligations cannot be fulfilled 'in a comprehensible manner'. The principle of transparency must therefore be taken into account from the outset of AI development.

The aim of development should be to create 'explainable AI'. AI systems must be able to explain how they arrived at their results. Only then can the reasons for an AI decision be explicitly described and communicated to those affected in a comprehensible manner.

Accuracy and data quality

According to Art. 5 para. 1 lit. d GDPR, personal data must be factually correct and, if necessary, kept up to date. This requires a comprehensive quality check of the training datasets used. The AI Act also stipulates comparable quality requirements for training, validation, and test datasets in Art. 10(3). 3. Better quality data leads to better AI results, because AI systems are only as ‘intelligent’ as their training data allows.

Confidentiality

When processing personal data, its confidentiality must always be guaranteed. If an AI system is continuously trained with interaction and user-specific raw data that is generated during operation, it must be ensured that this data is only processed at a local level if possible and deleted after training. If this is not possible, confidentiality must be ensured through end-to-end encryption. On the server side, the data may only be processed by authorised persons.

What data protection measures must be taken when using AI?

Invest in AI literacy: Use training, guidelines and discussions to sensitise employees and stakeholders to AI ethics and governance.

• Invest in AI literacy: Raise awareness of AI ethics and governance among employees and stakeholders through training, guidelines and discussions.
• Ensure data quality and integrity: Ensure that all collected data is accurate and up to date. Review and update data regularly.
• Data access and security controls: Restrict access to data to employees who require it for their job role. Implement robust security measures, such as encryption, firewalls, and regular security updates, to protect data from unauthorised access and cyberattacks.
• Data Protection Impact Assessment (DPIA) and Risk Assessment: Conduct a DPIA for particularly high-risk processing operations to assess the impact on data protection and take appropriate measures. Conduct regular risk assessments to identify and minimise potential data protection risks.

How can we help you use AI in a way that complies with data protection regulations?

Schürmann Rosenthal Dreyer is a law firm that specialises in technology law. We can develop a comprehensive AI compliance strategy for your company that includes data protection.