Pyspark brings together the analytical power and popularity of Python with the distributed-computing capability of Spark. In this post I show how you can use a docker container with pyspark and spark pre-loaded to let you play with pyspark in a Jupyter notebook, rather than having to configure your own spark cluster first. Use Jupyter… Read More »
The CoViD-19 global pandemic has thrust modelling and analysis into the public eye in a way rarely seen before. One particular example is the CoViD model developed by MRC-IDE, which was a contributor to the UK response to the pandemic. The research team behind this model published their code on GitHub. As might be expected… Read More »
NiFi is a tool designed to support the flow of data between software systems. This post shows how you can achieve NiFi version control with NiFi Register and docker. A note of caution: this post is about showing a little of what of possible with NiFi version control, and I suggest you read the NiFi… Read More »
Once you have trained a sci-kit learn model it is not obvious how you can deploy it and use it to score unseen data. This post shows you how to save and learn sci-kit learn models so you can execute it against unseen data. Train Your Model The first step is to train the model… Read More »
Matplotlib is the excellent workhorse plotting library for python. As great as matplotlib is, it could occasionally do with a little stylistic improvement [1] [2]. I recently came across a cool feature in matplotlib that lets you plot in the style of the popular XKCD comics, with a fun font and a more ‘sketchy’ line… Read More »