Blog
#Python
Discover our articles on IT performance, cloud, data science and industry best practices.
All articles
Cycle diurne de la température d'été à Lyon-Bron, par décennie
Météo-France publie ses données climatologiques de base horaires en open data sur data.gouv.fr.
Summation of Floating-Point Numbers in Cython
Adding up a list of floating-point numbers: seems simple. IEEE 754 double-precision floats (float64) carry about 15 to 17 significant decimal digits, and…
An example ETL Pipeline with dlt + SQLMesh + DuckDB
In this post, we walk through building a basic ETL (Extract-Transform-Load) pipeline.
A Git commit temporal analysis
In this Python notebook, we are going to analyze git commit timestamps across multiple repositories to identify temporal patterns in a git user coding…
Computing and Visualizing Billions of Bohemian Eigenvalues with Python
In our case, we sample 5x5 matrix entries from the discrete set {-1, 0, 1}.
Zipf's Law on "La Comédie humaine"
In this Python notebook, we're going to explore Zipf's law as applied to words. This is the definition of Zipf's law from its wikipedia page.
Arnold tongues with Numba, Numba CUDA and Datashader
This Python notebook explores Arnold tongues. Here is a short description from wikipedia: In simpler terms, Arnold tongues represent regions where two…
Spatial queries in DuckDB with R-tree and H3 indexing
This Python notebook explores the use of two indexing techniques, R-tree and H3, for performing spatial queries in DuckDB.
In-memory cosine distance calculation, NumPy vs DuckDB
This notebook compares the performance of NumPy and DuckDB for computing cosine distances on the same dataset.
Pizzerias Around the World
In November 2024, Foursquare released a new dataset called Foursquare Open Source Places.
Visualizing the Ulam Spiral with Python
The Ulam spiral arranges integers in a spiral and highlights prime numbers, revealing some diagonal alignment patterns.
Analyzing tramontane wind frequency
The tramontane wind, a defining feature of Pyrénées-Orientales region, is known for sweeping through the south of France with its characteristic force and…
Finding the most concentrated areas of bakeries in Lyon
In this blog post, we'll use Python to analyze the distribution of bakeries in Lyon, France.
A Hybrid information retriever with DuckDB
When it comes to information retrieval, vector search methods have demonstrated some good performance, especially when the embedding models have been…
Enrich a digital terrain model
In this blog post, we will explore how to enrich a digital terrain model [DTM] using Python.
Calculating walking isochrones with Python
In this blog post, we'll explore how to calculate walking isochrones using Python, taking into account the slope of the terrain.
Fully Amortized Loan simulation with Numba and IPyWidgets
In this blog post, we will show how to use Python to simulate the amortization of a fully amortized loan, such as a mortgage or a car loan.
Create a routable pedestrian network with elevation
In this blog post, we will explore how to create a routable pedestrian network with elevation using Python and OSMnx.
Streaming data from PostgreSQL to a CSV file as fast as possible
In this post, we explore the process of streaming data from a PostgreSQL database to a CSV file using Python.
Lyon's Digital Terrain Model with IGN Data
In this post, we explore how to extract and merge data from a french high-resolution Digital Terrain Model [DTM].
A Transit graph for static macro assignment
This post is a description of a graph structure for a transit network, used for static, link-based, frequency-based assignment.
Parquet file sorting test
Some time ago, we came across an intriguing Parquet sorting test shared by Mimoune Djouallah on Twitter @mim_djo.
Installing your Python package on a Windows machine that does not have internet access
Suppose you've developed a Python package called MyPackage on Linux, with specific package requirements, and need to install it on a Windows machine that…
Calculating daily mean temperatures with scikit-learn
The goal is of this post is to predict the daily mean air temperature TAVG from the following climate data variables: maximum and minimum daily…
Vector similarity search with pgvector
In the realm of vector databases, pgvector emerges as a noteworthy open-source extension tailored for Postgres databases.
Using a local sentence embedding model for similarity calculation
A simple yet powerful use case of sentence embeddings is computing the similarity between different sentences.
Python plot - Antarctic sea ice extent
Data source: https://ads.nipr.ac.jp/vishop/#/extent REGION SELECTOR = Antarctic At the bottom of the page: Download the sea ice extent (CSV file) -…
Python plot - North Atlantic daily water surface temperature
Data source: https://climatereanalyzer.org [NOAA Optimum Interpolation SST [OISST] dataset version 2.1] In the present dataset, the surface temperature is…
Hyperpath routing in the context of transit assignment
How do transit passengers choose their routes in a complex network of lines and services?
TPC-H benchmark of DuckDB and Hyper on native files
In this blog post, we examine the performance of two popular SQL engines for querying large files: These engines have gained popularity due to their…
TPC-H benchmark of Hyper and DuckDB on Windows and Linux OS
In this blog post, we explore the use of two SQL engines, and specifically their Python API, for querying files.
TPC-H benchmark of Hyper, DuckDB and Datafusion on Parquet files
In this blog post, we focus on directly querying Parquet files using three different SQL engines, and more specifically their Python API: The TPC-H…
Dijkstra's algorithm in Cython, part 3/3
Running time of Dijkstra's algorithm on DIMACS networks with various implementations in Python.
Dijkstra's algorithm in Cython, part 1/3
In this post, we are going to present an implementation of Dijkstra's algorithm in Cython. Dijkstra's algorithm is a shortest path algorithm.
Dijkstra's algorithm in Cython, part 2/3
This post is the second part of a three-part series. In the first part, we looked at the Cython implementation of Dijkstra's algorithm.
A Cython implementation of a priority queue
Credit: Musée de l'illusion, Lyon [picture taken by myself] In this post, we describe a basic Cython implementation of a priority queue.
Visualizing some polynomial roots with Datashader
Last week-end I found this interesting tweet by sara: The above figure shows all the complex roots from the various polynomials of degree 10 with…
Forward and reverse stars in Cython
This notebook is the following of a previous one, where we looked at the forward and reverse star representations of a sparse directed graph in pure…
Forward and reverse star representation of a digraph
In this Python notebook, we are going to focus on a graph representation of directed graphs: the forward star representation [and its opposite, the…
Query Parquet files with DuckDB and Tableau Hyper engines
In this notebook, we are going to query some Parquet files with the following SQL engines: Both of these tools are optimized for Online analytical…
Download some benchmark road networks for Shortest Paths algorithms
The goal of this Python notebook is to download and prepare a suite of benchmark networks for some shortest path algorithms.
Euler's number and the uniform sum distribution
Last year I stumbled upon this tweet from @fermatslibrary: I find it a little bit intriguing for Euler's number e to appear here!
Trying DuckDB with Discogs data
This notebook is a small example of using DuckDB with the Python API. It is a relational DBMS that supports SQL.
Dynamic TaskGroup in Airflow 2.0
In this article we will uncover a way to use Airflow new feature called TaskGroup which allow you to manage your dependencies in a dynamic way.
Dynamic TaskGroup Scalability in Airflow 2.0
In the previous article I showed you how to instantiate TaskGroup in a Dynamic way.
Reading a SQL table by chunks with Pandas
In this short Python notebook, we want to load a table from a relational database and write it into a CSV file.
Apache Airflow 2.0 : How it works
Airflow became in the last recent years a major actor for scheduling a wide variety of actions.
Loading data from CSV files into a Tableau Hyper extract
Hyper is Tableau’s in-memory data engine technology, designed for fast data ingest and analytical query processing on large or complex data sets.
More Heapsort in Cython
This post/notebook is the follow-up to a recent one: Heapsort with Numba and Cython, where we implemented heapsort in Python/Numba and Cython and compared…
Loading data from PostgreSQL to Pandas with ConnectorX
ConnectorX is a library, written in Rust, that enables fast and memory-efficient data loading from various databases to different dataframes.
Export data as fast as possible : from HANA to CSV
I want to export from HANA to CSV. As source, a table or a sql query, as target an external client, of course as fast as possible and using a command line…
Heapsort with Numba and Cython
Heapsort is a classical sorting algorithm. We are going into a little bit of theory about the algorithm, but refer to Corman et al.
Using Tableau to detect outliers and ruptures
To do that, we are going to get the outliers of a dataset, the change points and a piecewise approximation.
Plotting population density with datashader
In this short post, we are using the Global Human Settlement Layer from the European Commission: The downloaded file has a worldwide resolution of 250m…
Applying a row-wise function to a Pandas dataframe
More than 3 years ago, we posted a comparative study about Looping over Pandas data using a CPU.
A Parallel loop in Python with Joblib.Parallel
The goal of this post is to perform an embarrassingly parallel loop in Python, with the same code running on different platforms [Linux and Windows].
Python Spatial Join with GeoPandas (and GEOS)
The purpose of this post is to perform an "efficient" spatial join in Python. What is a spatial join?
Built-in Expectations in Great Expectations
Great expectation is a Python tool for data testing, documentation, and profiling.
Goldbach's Comet with Numba and Datashader
This Python notebook is about computing and plotting Goldbach function. It requires some basic mathematical knowledge, nothing fancy!
Le tour de france history web scraping
This file downloads raw data about every rider of every Tour de France (from 1903 up to 2020).
Some Pre-commit git hooks for Python
Pre-commit hooks are a great way to automatically check and clean the code. They are executed when committing changes to git.
Quick data exploration with pandas, matplotlib and seaborn
In this JupyterLab Python notebook we are going to look at the rate of coronavirus [COVID-19] cases in french departments [administrative divisions of…
Optuna and XGBoost on a tabular dataset
The purpose of this Python notebook is to give a simple example of hyperparameter optimization [HPO] using Optuna and XGBoost.
Saving a tf.keras model with data normalization
Training a DL model might take some time, and make use of some special hardware.
Benford's law and the population of french cities
In this Python notebook, we are going to look at Benford's law, which predicts the leading digit distribution, when dealing with some real-world…
Merge Sort with Cython and Numba
In this post, we present an implementation of the classic merge sort algorithm in Python on NumPy arrays, and make it run reasonably "fast" using Cython…
Logistic regression with JAX
JAX is a Python package for automatic differentiation from Google Research. It is a really powerful and efficient library.
Minimizing continuous non-convex functions with Optuna
In this post, we are going to deal with single-objective continuous optimization problems, using the open-source Optuna Python package.
Lunch break, fetching AROME temperature forecast in Lyon
Since a "small" heat wave is coming, I would like to get some temperature forecast for the next hours in my neighborhood, from my JupyterLab notebook.
Lunch break, ridge plots with Bokeh
Bokeh is a great visualization Python library. In this short post, we are going to use it to create a ridge plot.
Outlier and Change Point Detection in Data Integration Performance Metrics
Data integration involves combining data residing in different sources, and providing users with a unified view of them.
Lunch break, plotting excess death in french department zones with Python
Daily deaths data are provided by INSEE - the national institute of statistics and economic studies.
A Quick study of air quality in Lyon with Python
The aim of this post is to use Python to fetch air quality data from a web service and to create a few plots.
Fitting a logistic curve to time series in Python
In this notebook we are going to fit a logistic curve to time series stored in Pandas, using a simple linear regression from scikit-learn to find the…
Cython and Numba applied to a simple algorithm: Insertion sort
The aim of this notebook is to show a basic example of Cython and Numba, applied to a simple algorithm: Insertion sort.
Lunch break: plotting traffic injuries with datashader
Well I love the datashader Python package and I am always happy to use it on some interesting datasets.
Fetching AROME weather forecasts and plotting temperatures
Accurate weather forecasts might be very usefull for various types of models.
Some cool open-source Python packages for Machine Learning Ep 4
There is a very rich ecosystem of Python libraries related to ML.
Some cool open-source Python packages for Machine Learning Ep 3
There is a very rich ecosystem of Python libraries related to ML.
First try of auto-sklearn
Since we are big users of scikit-learn and XGBoost, we wanted to try a package that would automate the process of building a machine learning model with…
Some cool open-source Python packages for Machine Learning Ep 2
There is a very rich ecosystem of Python libraries related to ML.
Loading data into a Pandas DataFrame - a performance study
Because doing machine learning implies trying many options and algorithms with different parameters, from data cleaning to model validation, the Python…
Some cool open-source Python packages for Machine Learning Ep 1
There is a very rich ecosystem of Python libraries related to ML.
GPU Analytics Ep 3, Apply a function to the rows of a dataframe
The goal of this post is to compare the execution time between Pandas (CPU) and RAPIDS (GPU) dataframes, when applying a simple mathematical function to…
Nighttime Lights with Rasterio and Datashader
In this post, we are going to plot some satellite GeoTIFF data in Python.
Symmetric Chaos with Datashader and Numba
Map equation and coefficient values are taken from here. Some mathematical explainations can be found here, by Mike Field and Martin Golubitsky.
Plotting Hopalong attractor with Datashader and Numba
What is an attractor? Definition from wikipedia: Most of the following code comes from James Bednar's notebook about 2D strange attractor plotting with…
Looping over Pandas data
I recently stumbled on this interesting post on RealPython (excellent website by the way!): Fast, Flexible, Easy and Intuitive: How to Speed Up Your…
Pandas Time Series example with some historical land temperatures
The aim of this notebook is just to play with time series along with a couple of statistical and plotting libraries.