Natural Language Processing Basics
NLP turns text into numbers so models can learn from it.
Pipeline
raw text -> clean -> tokenise -> vectorise -> model -> prediction
Preprocessing
import re
def clean(text: str) -> str:
text = text.lower()
text = re.sub(r"http\S+", " ", text) # remove URLs
text = re.sub(r"[^a-z0-9\s]", " ", text) # keep letters and digits
return re.sub(r"\s+", " ", text).strip()
print(clean("Check https://x.io: PrepVerse is GREAT!!!"))
# check prepverse is great
Typical steps: lowercasing, tokenisation, stop-word removal, stemming or lemmatisation. Modern transformer models usually need little preprocessing.
Bag of words and TF-IDF
| Representation | Idea | Limitation |
|---|---|---|
| Bag of words | Count of each word | Ignores order and meaning |
| TF-IDF | Down-weights words common to all documents | Still sparse, no semantics |
| Embeddings | Dense vectors where similar words are close | Needs training data |
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
texts = ["great course, learned a lot", "terrible, waste of time",
"excellent explanations", "boring and confusing"]
labels = [1, 0, 1, 0]
model = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)), LogisticRegression())
model.fit(texts, labels)
print(model.predict(["very clear and helpful"])) # [1]
Embeddings and transformers
- Word embeddings (word2vec, GloVe): one vector per word.
- Contextual embeddings (BERT and relatives): the vector depends on the sentence.
- Transformers use self-attention to relate every token to every other token; large language models are transformers trained on huge text corpora.
from transformers import pipeline
clf = pipeline("sentiment-analysis")
print(clf("The lessons are clear and well structured."))
Common tasks
| Task | Example |
|---|---|
| Classification | Spam detection, sentiment |
| Named entity recognition | Find people, places, dates |
| Summarisation | Short abstract of an article |
| Question answering | Answer from a passage |
| Machine translation | English to Hindi |
CAUTION
Evaluate on realistic data, check for bias, and never assume a model's fluent output is correct.
Loading Comments...