In this post, I overview the optimal non-parametric approach to quantile confidence intervals. We will discuss the theoretical background as well as an efficient algorithm implemented in Python. Consider an ordered iid sample from a distribution , and let be the -th quantile, that is . Using our samples we want to build a possibly …
Continue reading “Correct Confidence Intervals for Quantiles”
The zero-truncated Poisson distribution is useful to model certain demographic data, such as sizes of households and families 1. This post illustrates this on historical data from the United Kingdom and demonstrates how to assess the goodness-of-fit in Python. The zero-truncated Poisson distribution is defined as\begin{align}\mathbf{P}\{Y=x\} = \mathbf{P}\{ X = x | X > 0\}, …
Continue reading “Demographic Data with Zero-Truncated Poisson”
QQ plots, despite their importance, are not well-supported in Python, as of the moment of writing this note. The package statsmodels offers plots of rather poor quality and non-interactive, and even Plotly doesn’t come with its own recipe. In this note, I am sharing how to build an interactive QQ plot and apply it to …
Continue reading “Diagnosing regression with interactive QQ plots”
Normality of data is important in statistics, so it is not a surprise that transforming data to look closer to a normal distribution can be beneficial in many ways. There are ways of finding the right transform, such as the popular Box-Cox method 1,2. In this post, I will share an elegant example on how …
Continue reading “On Importance of Log-Normality in Hypothesis Testing”
Two-player games can be solved by following a very intuitive algorithm called Regret Matching 1. Players modify their action probabilities according to the so-called regrets or advantages, which can be thought as consequences of alternative choices. For a good overview of the topic, see the friendly yet detailed introduction by Neller and Lanctot 2. The …
Continue reading “Approximate Nash Equilibrium by Regret Matching”
This post explains how to measure GPU utilisation in real-time using tools provided by NVIDIA.
Staring from the version 2.x PyTorch, a popular deep-learning framework, introduces a JIT compiler torch.compile. In this post, I am sharing a non-trivial example demonstrating how this tool can reduce memory footprint on GPU. The point of departure is a sub-routine which computes similarity, similar to covariance but not as friendly to compute. For two …
Continue reading “Compiling Torch Code – Example on Memory Reduction”
Location on maps are often provided in a Coordinate Reference System (CRS). In computer vision projects, however, we need to translate them to pixels. Plotting and transformations can be accomplished with the rasterio Python module. Below I share a useful snippet showing how to convert CRS locations to pixel coordinates:
Visualizing code as a syntax tree is both funny and useful, as seen from impressive applications such as creating lineage of SQL which helps to understand complex queries in business. Abstract syntax trees are not only widely used in industry but are still a subject of top academic research1,2. This post demonstrates how to work …
How to prototype with multiple ML libraries in Cloud? Best to build on top of a rich pre-configured environment such as Kaggle image, extending it with a local virtual environment.
