Airflow is a workflow management platform for building data engineering pipelines. It uses Python to define your data engineering workflow as code, making them more maintainable, versionable, testable and collaborative. In this article, I will try to explain a few important concepts and terminology associated with Airflow. DAGs A DAG…

If you are starting out in any branch of computer science, whether software development, data science, software, and data engineering, should at the least become familiar with basic Linux commands. Here I am listing a few of them: man Short for ‘manual’: it provides the details about any command in…

Some of the most common metacharacters used in Regex along with usage: . matches any character including whitespace. + matches 1 or more occurrences of the preceding subexpression. ? matches 0 or 1 occurrence of the preceding subexpression. * matches 0 or more occurrences of the preceding subexpression.

Data Normalisation is a bottom-up technique for database design, as opposed to Entity-Relationship Diagrams, which are a top-down technique for the same. These two approaches generally complement each other, and a balanced approach to database design will use both these approaches. Data normalisation is the technique of organising data into…

In this article, I have tried to summarise the main features of the most common supervised ML algorithms in use, excluding ensemble methods such as RandomForest or XGboost, which I will deal with in another article. Please note that in all the cases below, n represents the number of input…

In the previous article, I discussed building a linear regression model using Tensorflow. In this article, I will try to solve a multiclass classification problem using Tensorflow. I have used the MNIST-digit recognizer dataset here. Please note that even though a Convolutional Neural Network might have worked better for this…

A neural network is normally associated with Deep Learning problems, such as Image classification or Natural Language Processing. But it can as easily be trained to build a Linear Regression model. In fact, building a Linear Regression model is a good starting point for learning how to build neural networks. …

A decision boundary is a surface that separates two or more classes into different sets, where all the points belonging to one class lie on one side of the decision boundary. Plotting a decision boundary is a great way to visually evaluate how good our machine learning model is, and…

Inferential statistics, where we take a sample from a population and use that sample to make predictions about the populations, is all about Hypothesis testing. For example, suppose we have been told that the mean height of 25-year old males in the US is 172 cm(=𝜇). We could try to…