Building Data Pipelines with Kedro

Kedro, an open-source Python framework, helps data science teams create maintainable data pipelines.

12 minutes Ben Hayes

Folks have compared data to oil in the past, and while the metaphor may be tired, the process of building pipelines in both cases consumes valuable time and resources, and often results in a mess. Read more to learn about Kedro!

Getting Started with Databricks

Databricks is the Data + AI company and the provider of a leading data science and machine learning platform.

6 minutes Ben Hayes

Ingesting, cleaning, and analyzing data on Databricks accelerates your data science and machine learning projects by leveraging the underlying data lake and Spark engine. Continue reading to get started with Databricks.

Exploring Probability Distributions

Probability distributions are used throughout quantitative fields where uncertainty or randomness are involved.

22 minutes Ben Hayes

Probabilistic thinking requires knowledge of probability distributions and the relevant statistics associated with each. Understanding these distributions can help you identify opportunities to leverage one. Continue reading to learn about the core concepts and essential distributions with examples in Python.

Time Series Analysis with Facebook Prophet

Time series analysis is a powerful technique to generate forecasts with seasonal data.

11 minutes Ben Hayes

Data sets contain noise but with high-powered or elegant data science, the relevant signal can be extracted. One key technique for analysis of real-world data (primarily focused on forecasting) is time-series analysis. A popular time-series forecasting procedure is Facebook's open-source Prophet procedure. Prophet is implemented in both R and Python.

Thank you, Carnegie Mellon University!

An expression of gratitude and summary of experience from one of Carnegie Mellon's latest graduates.

10 minutes Ben Hayes

Carnegie Mellon University's Heinz College offered a unique opportunity to jointly study public policy, management, and data analytics. The coursework covers topics ranging from machine learning, deep learning, econometrics, and optimization to organizational design and decision-making. Continue reading to learn why I am glad I enrolled.

What's Cooking? Cuisine Classification

Using only ingredients, learn how to classify recipes into cuisines using Python, classification methods, clustering methods, and Sci-kit Learn.

14 minutes Ben Hayes

While seemingly a trivial task, classifying recipes into cuisines and understanding how to interpret clustering and classification results can help you creatively answer other questions. Continue reading to learn how.

Analyzing Suspicious IP Addresses

Suspicious connection requests are a common occurrence for Linux system administrators. From who and where are these suspicious requests originating?

13 minutes Ben Hayes

Cyber attacks are becoming more common as more data are stored in digital form. Suspicious, and likely malicious, users are requesting access to unauthorized resources while hoping for vulnerable networks or systems. Continue reading to learn how to apply descriptive analytics to uncover who is generating these suspicious requests.

Introduction to Deep Learning with Python

From concept to code: introduce yourself to neural networks and deep learning.

11 minutes Ben Hayes

Neural networks are a growing area of research and are being applied to new problems every day. The classic examples are image classification, facial recognition, and self-driving cars. The future is uncertain, but there is a high probability that neural networks in some form will play a critical role in shaping it. Continue reading to learn about the core concepts and walk through an example in Python.

Building Shiny Dashboards

Learn how to build a shiny dashboard in R to help users analyze, visualize, and understand their data.

26 minutes Ben Hayes

Shiny dashboards provide a simple and fast way to analyze and visualize data. Whether performing exploratory data analysis or building a robust tool for your client's executives, Shiny dashboards aid the data science process. Continue reading to walk through an example of constructing an R Shiny Dashboard.

ACLU Data Science Internship

Data science for good: my internship with the ACLU in NYC

7 minutes Ben Hayes

Data science can change how any organization operates not just Facebook or Amazon, but even the American Civil Liberties Union - an organization 10 times older than Facebook. Continue reading to learn about my experience interning with the ACLU's data science team in New York.

Predicting Criminal Recidivism with R

Can data science indicate what factors affect the rate of criminal or violent recidivism? (Hint: Yes)

45 minutes Ben Hayes

Around the United States, municipalities have turned to risk assessment instruments (RAIs) to help judges determine which individuals to release on bail and which ones to keep in custody. The risk assessment process varies based on the specific instrument used but many rely on criminal recidivism data sets. These data sets typically contain various demographic indicators (age, race, gender, etc.) and also criminal history (charges, juvenile record, etc.).

Broward County, Florida, has turned to the use of one of the most popular RAIs today: COMPAS or the Correctional Offender Management Profiling for Alternative Sanctions tool. COMPAS assesses individuals based on criminal history and social profiling to categorize an individual as low, medium, or high risk. This tool, however, was not developed using the Broward County data set which may lead to poor performing predictions for individuals from Broward County, Florida. In this post we construct an RAI, compare to COMPAS and discuss findings.

The Gender Income Gap

How do factors (besides gender) affect the gender income gap?

20 minutes Ben Hayes

The issue of gender-income disparity is not new - it is nuanced. In order to answer these questions, we will rely on the National Longitudinal Survey of Youth, 1979 cohort, data set (abbreviated ‘NLSY79’). To draw strong conclusions, we must evaluate the data set provided - is it accurate, relevant, and useful for drawing statistical conclusions? Once we summarize the data, we can discuss methodology - how should we approach the data, what variables should we consider, what techniques are appropriate? Third, we openly discuss findings about the sampled individuals and attempt to infer relationships about the income difference (if any) between men and women in the larger population based on other factors. Lastly, we end with a discussion of the relevancy and signicance of our findings given the context of the survey data available and the methodology used.

Demystifying the Confusion Matrix

A simple 2x2 table can contain insightful metrics that enhance your decision-making.

7 minutes Ben Hayes

Understanding the confusion matrix is an important step in statistics, machine learning, or any other field where predictions or classifications are common. The confusion matrix is a type of contingency table with two dimensions that reveal how well a predictive model performs when the outcomes are known. Additionally, when associated costs of incorrect positive and negative guesses differ, the trade-offs can be optimized. Do you know the difference between Sensitivity, Specificity, Recall, Precision, True Positive Rate, and Positive Predictive Value?