Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 76 ยท What Data Science Is

Python Libraries for Data Science, ML and Deep Learning

A map of the Python data ecosystem: arrays, tables, statistics, plotting, machine learning, deep learning, NLP, big data and tooling, with one runnable demo per core library.

Python is the main language of data science mostly because of its libraries. A handful of them do the same jobs in almost every project: store numbers, hold tables, compute statistics, draw charts, train models. Hundreds more specialize: language processing, images, graphs, time series, big data, deployment. Knowing what exists, and which tool to reach for first, saves you from rewriting what a mature library already does well.

This lesson is a map, not a manual. It groups the libraries by the job they do, shows one tiny runnable example for each core one, and ends with advice for choosing and keeping up to date. The rest of this course teaches the core libraries in depth. If you are still setting up, see The Data Science Workflow for how the tools fit together.

The core stack

The core stack

NumPy Fast arrays
pandas Tables and time series
Matplotlib Charts you control
scikit-learn Classic machine learning

Almost every other library builds on these.

1 / 2
  • NumPy provides the ndarray, a typed block of numbers with vectorized math, broadcasting and linear algebra. Nearly everything else stores its numbers in NumPy arrays.
  • pandas adds labelled tables (DataFrame) and series with grouping, joining, reshaping, dates and missing-value handling. It is where most data work happens.
  • SciPy supplies scientific routines: statistics and distributions, optimization, interpolation, signal processing, sparse matrices, clustering.
  • statsmodels fits statistical models and reports what a statistician wants: coefficients, standard errors, confidence intervals and p-values.
  • Matplotlib is the base plotting library; Seaborn adds attractive statistical charts on top of it, and Plotly makes interactive ones for the web.
  • scikit-learn offers a consistent interface (fit, predict, transform) for classification, regression, clustering, dimensionality reduction, preprocessing, model selection and pipelines.
Python One call from each core library

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All four tools recover roughly the line that generated the data (intercept near 3, slope near 2); they differ in what they give you around the answer. scikit-learn is built for prediction, statsmodels for inference.

Visualization

Draw first with pandas' built-in .plot, then move to the right tool:

  • Matplotlib gives full control over every element; see Matplotlib for Analysts.
  • Seaborn produces distribution plots, heatmaps and grouped comparisons from a DataFrame in one line; see Seaborn for Analysts.
  • Plotly and Altair create interactive or declarative charts; see Plotly for Analysts.
  • Other names you will meet: Bokeh (interactive web plots), Dash, Streamlit and Panel (turn analyses into apps), Folium (maps).

Machine learning beyond scikit-learn

  • XGBoost, LightGBM and CatBoost are gradient-boosting libraries that often win on tabular data (see Tree Models and Ensembles).
  • imbalanced-learn adds resampling methods such as SMOTE for rare classes.
  • MLflow and DVC track experiments, models and data versions.
  • Optuna and scikit-learn's own search tools tune hyperparameters.
  • SHAP and LIME explain individual predictions (see Model Interpretation).

Deep learning

Deep learning libraries compute gradients automatically and run on GPUs. The big three are PyTorch (popular in research and increasingly in production), TensorFlow with Keras (a high-level model-building API) and JAX (functional, compiled array programming). Related tools: Hugging Face Transformers (pretrained language, vision and audio models), PyTorch Lightning (training boilerplate) and ONNX (exchange format for deployment). Most tabular problems do not need them; pictures, sound and text usually do.

Text, images and other data types

  • NLP: NLTK (teaching and classic tools), spaCy (fast production pipelines), Gensim (topic models and embeddings), Transformers (modern models). See Text Data Basics.
  • Computer vision: OpenCV (image processing), Pillow (loading and editing images), scikit-image, torchvision.
  • Time series: statsmodels, Prophet, sktime, Darts. See Time Series Forecasting.
  • Graphs and networks: NetworkX for analysis, PyTorch Geometric for graph neural networks.
  • Geospatial: GeoPandas, Shapely, Folium.

Data access, big data and tooling

  • Getting data: requests and httpx for APIs, Beautiful Soup and Scrapy for scraping, SQLAlchemy and DuckDB for databases (see SQL for Data Science and Web Scraping and APIs).
  • Bigger than memory: Polars (a fast DataFrame library with lazy evaluation), Dask and PySpark (distributed processing), DuckDB (SQL on files), PyArrow (columnar format shared by many tools).
  • Notebooks and environments: Jupyter, VS Code, venv, uv, conda. See Reproducible Projects.
  • Quality and testing: pytest, Great Expectations and Pandera for data checks, pyjanitor for cleaning verbs (see Text and Category Cleaning).

Choosing and trusting a library

  1. Start with the standard stack. NumPy, pandas, Matplotlib and scikit-learn solve most problems; a specialised library should earn its place.
  2. Check that it is maintained: recent releases, an active issue tracker, documentation with examples, and a license that fits your use.
  3. Pin versions. Libraries change behavior between releases (a method is renamed, a default flips). Record exact versions in requirements.txt or a lock file and print __version__ in your notebook.
  4. Read the release notes when you upgrade; most breakages are listed there.
  5. Prefer a well-known API over a clever one. Colleagues (and you in six months) must be able to read the code.
Python Record the versions your analysis used

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which library provides the labelled DataFrame used for cleaning, grouping and joining tables?

    pandas builds labelled tables on top of NumPy arrays. NumPy supplies the arrays, Matplotlib draws, SciPy provides scientific routines.
  2. 2. You need coefficients with standard errors and p-values for a regression. Which library is built for that?

    statsmodels focuses on statistical inference and reports the quantities a statistician expects. scikit-learn focuses on prediction.
  3. 3. Which tools are gradient-boosting libraries often used on tabular data?

    XGBoost, LightGBM and CatBoost implement boosted decision trees, which are strong on structured data.
  4. 4. Why should you record library versions with your analysis?

    Pinned versions let you or a colleague recreate the same environment and get the same results later.

Summary

  • The core stack is NumPy (arrays), pandas (tables), Matplotlib/Seaborn/Plotly (charts), SciPy and statsmodels (statistics) and scikit-learn (machine learning).
  • Gradient-boosting libraries, PyTorch/TensorFlow/JAX, NLP, vision, time-series and geospatial libraries extend the core for specific problems.
  • Polars, Dask, PySpark and DuckDB handle data that is too big or too slow for plain pandas.
  • Choose maintained, well-documented libraries, start simple, and pin and record versions.

Next: Framing the Question.