Distributed Computing for Health Data Course on

Distributed Computing for Health Data - 10 - Kafka Basic Usage

Gonçalo Leiria — Fri, 30 Jan 2026 06:29:24 +0000

These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.

Lesson 10 - Kafka Basic Usage

In the first half of the lesson, they follow Kafkas’s quickstart tutorial so I won’t be copying and pasting it here. Instead, since I had the need to simplify core concepts and build a mental model of how Kafka actually works I slightly deviated from the the first half of the lesson.

Distributed Computing for Health Data - 9 - Kafka

Gonçalo Leiria — Wed, 28 Jan 2026 14:50:29 +0000

These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.

Lesson 9 - Kafka

So far we spent the course learning how to process data at rest. Kafka is about data in motion. Here we look into Kafka’s architecture and interface.

Concepts

Event Streaming - characteristics

Event: an immutable object or fact of interest

Distributed Computing for Health Data - 8 - Spark Basic Usage

Gonçalo Leiria — Thu, 22 Jan 2026 07:42:49 +0000

These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.

Lesson 8 - Spark Basic Usage

In this lesson I follow the main core ideas of the course, but I deviate a bit from the way it is taught. The instructors use a pre-written PySpark Jupyter notebook and execute the workflow cell by cell. I prefer to work from VScode + terminal so created the following file strucutre:

Distributed Computing for Health Data - 7 - Spark SQL and DataFrames

Gonçalo Leiria — Tue, 20 Jan 2026 18:09:47 +0000

These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.

Lesson 7 - Spark SQL and DataFrames

In this lesson, we advance to higher level interfaces - SparkSQL and DataFrames - which allow for structured data manipulation in Spark. Remember that structured data is data that fits into a table. So structured data manipulation means filtering, selecting, grouping, joining, (etc) and aggregating data that is organised in rows and columns. Also, Spark turns every input data (for example JSON) into a DataFrame so it can optimises.

Distributed Computing for Health Data - 6 - Spark and Spark Components

Gonçalo Leiria — Sun, 18 Jan 2026 10:08:56 +0000

These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.

Lesson 6 - Spark and Spark Components

What is Spark

Software platform for data engineering, data science, and machine learninng
Designed to be fast and general purpose
Multi-language: Python, R, SQL, Java, Scala.

Spark vs Hadoop

See paragraph relevant section on lesson 5

Distributed Computing for Health Data - 5 - Hadoop Basic Usage

Gonçalo Leiria — Tue, 13 Jan 2026 07:25:55 +0000

These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.

Lesson 5 - Hadoop Basic Usage

In this lesson there is a lot of information on different options for installing Hadoop, how to navigate the GUI, and how to connect to a remote master node via SSH. Here, I’ll simply put down some notes that I think are important to retain and keep in mind. In the next lessons there will be more hands-on data manipulation using Spark and Hadoop.

Distributed Computing for Health Data - 4 - Hadoop

Gonçalo Leiria — Mon, 12 Jan 2026 06:41:44 +0000

These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.

Lesson 4 - Big Data Processing Tools: Hadoop

Hadoop

Framework for distributed processing of large datasets
Allows an application to scale up from a single computer to thousands of machines while implementing data distribution and fault tolerance
Primarily intended for batch processing
Hadoop is an ’ecosystem’ comprised of core and specific modules, each providing a different service
At its core, Hadoop has a distributed file system and a distributed processing layer, HDFS and MapReduce

Fault tolerance is the capability to continue operating smoothly despite failures or errors in one or more components of a distributed system. This resilience is crucial for maintaining system reliability, availability, and consistency.

Distributed Computing for Health Data - 3 - Distributed Computing

Gonçalo Leiria — Thu, 08 Jan 2026 14:05:28 +0000

These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.

Lesson 3 - Distributed Computing

Distributed Computing

Use of autonomous networked machines (nodes) for solving a single problem
Autonomous as each node only has access to its memory, processor and storage
No shared memory across nodes so they communicate via messages and carrying data
Data and processing tasks are distributed across nodes

Data Parallelism

Distributed Computing for Health Data - 2 - Big Data Sources

Gonçalo Leiria — Tue, 06 Jan 2026 15:03:10 +0000

These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.

Lesson 2 - Big Data Sources

Data Types and Data Sources

There are 5 data types:

Structured (e.g. Electronic Health Records)
Semi-structured ( e.g. EHR and vital-signs for a patien but not vital-signs for a second patient)
Unstructured (CT scan)
Time-series (ECG)
Sequence data (RNA or DNA sequencing for example)

Structured Data:

Distributed Computing for Health Data - 1 - Key Concepts

Gonçalo Leiria — Mon, 05 Jan 2026 13:31:45 +0000

These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.

Lesson 1 - Key Concepts

The 3 Vs of Big Data

A classic way to understand big data is through the 3 Vs:

Volume - how much data we have
Velocity - how fast data is generated and needs to be processed
Variety - how many different forms the data takes

In healthcare, all three Vs show up at once and that’s what makes the domain both challenging and powerful.