What is natural language processing (NLP)?

Definition

What is natural language processing?

Natural language processing (NLP) is a subfield of artificial intelligence that enables computers to analyse, understand and generate written and spoken language. NLP combines linguistics with machine learning to interpret texts and recognise intentions.

What does NLP mean in German?

The German translation of natural language processing is natürliche Sprachverarbeitung. The English term and the abbreviation NLP have become standard in professional use.

In IT and AI research, NLP stands exclusively for natural language processing. The term is sometimes confused with neuro-linguistic programming – a coaching method from psychology that has nothing to do with language processing by computers.

Which subfields does NLP include?

NLP is divided into two core areas: NLU (natural language understanding) and NLG (natural language generation). NLU enables machines to grasp the meaning and intent behind texts.

NLG generates coherent texts or spoken answers from that on its own. Together, both areas form the basis for applications such as chatbots, translation tools and voice assistants.

How it works

How does natural language processing work?

NLP processes language in several steps: the text is first broken down into individual parts, analysed grammatically and then interpreted for its meaning. This pipeline forms the backbone of all language processing.

Tokenisation and pre-processing

Every NLP process begins with tokenisation – breaking text down into individual words or parts of words (tokens). Stop words such as “and”, “the” or “is” are removed because they carry no substantive meaning.

Stemming reduces words to their stem: “running”, “ran” and “runs” are unified as “run”. Lemmatisation goes further and traces words back to their dictionary base form.

Syntactic and semantic analysis

POS tagging (part-of-speech tagging) determines the word class of each token – whether it is a verb, noun or adjective. Dependency parsing analyses the grammatical relationships between the words of a sentence.

The semantic analysis recognises meanings in context. It understands that “bank” in the sentence “I sit on the river bank” means a shore – and not a financial institution.

From rules to deep learning

The first generation of NLP systems worked with fixed if-then rules – severely limited and not scalable. Statistical methods brought the breakthrough from the 1990s by modelling language with probability models.

Since 2017, transformer models such as BERT (Google, 2018) and GPT (OpenAI), which process language via self-attention, have dominated. This architecture uses billions of parameters and has fundamentally changed the performance of NLP systems.

Practice

Which tasks does NLP solve in everyday life?

NLP is behind voice assistants, translation tools, chatbots and email filters – applications that billions of people use every day. The technology turns unstructured language data into structured information.

Voice assistants and chatbots

Alexa, Siri and Google Assistant convert spoken language into text, analyse the intent and generate a suitable answer. This three-step process – speech recognition, understanding, generation – is NLP in real time.

Customer servicechatbots use the same technique to classify enquiries automatically. Frequently asked questions are answered without human intervention.

Translation and text analysis

DeepL and Google Translate translate texts in real time between more than 100 languages. Both services use neural networks trained on millions of sentence pairs.

Sentiment analysis automatically detects whether a social media post, a product review or a comment is phrased positively, negatively or neutrally. Spam filters in email programs also classify incoming messages using NLP models.

NLP in companies

Companies use NLP for automatic document processing – for example to extract contract data, invoice items or customer data. Named entity recognition (NER) identifies people, places and organisations in texts.

The global NLP market reached a volume of $36.8 billion in 2025 and, according to Fortune Business Insights, is expected to grow to $193.4 billion by 2034.

Working with text and data

What role does NLP play in large language models?

Large language models such as GPT, Claude and Llama are based entirely on NLP methods – they are the most powerful application of natural language processing to date. Without NLP, these models could neither understand nor generate text.

From NLP pipelines to LLMs

Classic NLP systems process language in separate steps: tokenisation, feature extraction, model training. LLMs combine these steps in a single transformer architecture trained on billions of pieces of text data.

The basis remains the same: text is broken into tokens, encoded as embeddings (numerical vectors) and processed through several transformer layers. More on the page What are large language models.

Why LLMs build on NLP

Every input to an LLM goes through the same basic NLP steps: tokenisation breaks the prompt into tokens, embeddings turn them into vectors, and the transformer layers calculate relationships between all tokens.

Without this NLP pipeline, no LLM could process language. The difference from classic NLP systems lies in scale – LLMs use hundreds of billions of parameters instead of thousands.

ML context

How are NLP and machine learning connected?

NLP is one of the most important fields of application of machine learning – almost all modern NLP systems use ML algorithms to process language. Without machine learning, NLP would be limited to rigid rule sets.

NLP as a field of application of ML

Supervised learning trains NLP models with labelled data: a dataset of texts and associated categories teaches the model to classify new texts correctly. Unsupervised learning finds patterns without labels – for example in topic modelling, which recognises thematic clusters in large text collections.

A detailed introduction to the learning methods is provided in the article What is machine learning.

Training methods for NLP models

Self-supervised learning has changed NLP research: models such as BERT learn language structures by predicting masked words in sentences – without manually labelled data.

Transfer learning makes it possible to adapt a pre-trained language model to a specific task such as sentiment analysis or named entity recognition with little additional data.

Basis for prompts

Why is NLP the basis for prompt engineering?

Anyone who understands how NLP processes language formulates more precise prompts and achieves better results with AI models. The way a prompt is phrased directly influences the quality of the answer.

What tokenisation means for prompts

Every language model has a limited context window – measured in tokens, not words. A German word can span several tokens because tokenisation works at the subword level.

Anyone who knows this principle can phrase prompts more compactly and fit more context into the available window.

Better results through NLP understanding

NLP models respond to structure, context and word choice in the prompt. A clear task description with an explicit output format delivers more precise answers than a vague question.

More advanced techniques are described in the guide Learn prompt engineering.

FAQ

Frequently asked questions about natural language processing