"Hello world" version of an AI-data-analyst</h1> <article> <h1>"Hello world" version of an AI-data-analyst</h1> <p>Gonçalo Leiria — Mon, 10 Aug 2026 18:36:58 +0100</p> <p>I am developing an LLM-powered data analyst for a fictional SaaS company. The idea is that a user can ask questions about the business in natural language, and the agent can use the database to answer them if needed. I am using Pydantic-AI to connect the agent to Google’s Gemini models and Django to handle the application and data layer.</p> <p><strong>The SaaS Company</strong></p> <p>I created a fictional SaaS company and generated synthetic data for it. The goal is not to create a perfect simulation of a real business but to create a small coherent world that I can use to explore the AI-agent architecture and that the agent can reason about. As the project evolves, I will gradually add more data and relationships. This is literally just me putting the pieces together.</p> </article> <article> <h1>Django + Data Pipeline Deployment</h1> <p>Gonçalo Leiria — Wed, 22 Jul 2026 14:39:50 +0100</p> <h2 id="goal-of-the-project">Goal of the project</h2> <p>In my previous article I described how I built and deployed a Dockerised data pipeline to Microsoft Azure. Every day, Azure Data Factory orchestrates the execution of a container that collects weather data from the OpenWeather API and stores it as a Parquet file in Azure Blob Storage.</p> <p>The next step was to build a lightweight Django application that reads the latest dataset directly from Blob Storage and presents it through an interactive Plotly dashboard.</p> </article> <article> <h1>Blob Storage vs Azure Data Lake Storage Gen2</h1> <p>Gonçalo Leiria — Mon, 20 Jul 2026 07:03:06 +0100</p> <p>Recently, while deploying a <a href="https://github.com/gleiria/data-engineering-pipeline">data engineering project</a> to Azure, I found myself asking a question: <strong>when should I use Blob Storage and when should I use Azure Data Lake Storage Gen2 (ADLS Gen2)?</strong> Here I share how things clicked for me.</p> <h2 id="blob-storage">Blob Storage</h2> <p>Blob stands for <strong>Binary Large Object</strong>. A blob is simply a sequence of bytes. From the perspective of Azure Blob Storage, a JPEG image, a PDF, an MP4 video, a Parquet dataset, a CSV, or JSON file are all treated exactly the same: they are just blobs. Blob Storage is therefore Microsoft’s object storage service. It provides a highly scalable and inexpensive way of storing files in the cloud without caring about their internal structure.</p> </article> <article> <h1>A Simple Azure Deployment</h1> <p>Gonçalo Leiria — Thu, 16 Jul 2026 14:30:53 +0100</p> <p>This project started as a take-home assignment for a data engineering role. After, as part of my learning journey into cloud computing, and after completing Azure AZ-900 and DP-900 certifications, I wanted to get hands-on practice so I deployed the pipeline to Microsoft Azure. Here, I share the main steps I took to make that happen.</p> <p>A hypothetical team of data scientists asks me, as part of the data engineering team, to a build Dockerised data pipeline in Python to collect daily meteorological data from the publicly available OpenWeather API (<a href="https://openweathermap.org/)">https://openweathermap.org/)</a>. The GitHub repository for the project can be found <a href="https://github.com/gleiria/data-engineering-pipeline">here</a>. There you can find a detailed README describing the project’s architecture, design decisions, implementation, and instructions on how to run it locally. This post focuses exclusively on the deployment to Microsoft Azure.</p> </article> <article> <h1>Distributed Computing for Health Data - 10 - Kafka Basic Usage</h1> <p>Gonçalo Leiria — Fri, 30 Jan 2026 06:29:24 +0000</p> <p>These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.</p> <h3 id="lesson-10---kafka-basic-usage">Lesson 10 - Kafka Basic Usage</h3> <p>In the first half of the lesson, they follow Kafkas’s <a href="https://kafka.apache.org/quickstart/">quickstart tutorial</a> so I won’t be copying and pasting it here. Instead, since I had the need to simplify core concepts and build a mental model of how Kafka actually works I slightly deviated from the the first half of the lesson.</p> </article> <article> <h1>Distributed Computing for Health Data - 9 - Kafka</h1> <p>Gonçalo Leiria — Wed, 28 Jan 2026 14:50:29 +0000</p> <p>These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.</p> <h3 id="lesson-9---kafka">Lesson 9 - Kafka</h3> <p>So far we spent the course learning how to process data at rest. Kafka is about <strong>data in motion</strong>. Here we look into Kafka’s architecture and interface.</p> <hr> <h2 id="concepts">Concepts</h2> <p><strong>Event Streaming - characteristics</strong></p> <ul> <li> <p><strong>Event:</strong> an immutable object or fact of interest</p> </article> <article> <h1>Distributed Computing for Health Data - 8 - Spark Basic Usage</h1> <p>Gonçalo Leiria — Thu, 22 Jan 2026 07:42:49 +0000</p> <p>These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.</p> <h3 id="lesson-8---spark-basic-usage">Lesson 8 - Spark Basic Usage</h3> <p>In this lesson I follow the main core ideas of the course, but I deviate a bit from the way it is taught. The instructors use a pre-written PySpark Jupyter notebook and execute the workflow cell by cell. I prefer to work from VScode + terminal so created the following file strucutre:</p> </article> <article> <h1>Distributed Computing for Health Data - 7 - Spark SQL and DataFrames</h1> <p>Gonçalo Leiria — Tue, 20 Jan 2026 18:09:47 +0000</p> <p>These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.</p> <h3 id="lesson-7---spark-sql-and-dataframes">Lesson 7 - Spark SQL and DataFrames</h3> <p>In this lesson, we advance to higher level interfaces - SparkSQL and DataFrames - which allow for structured data manipulation in Spark. Remember that structured data is data that fits into a table. So structured data manipulation means filtering, selecting, grouping, joining, (etc) and aggregating data that is organised in rows and columns. Also, Spark turns every input data (for example JSON) into a DataFrame so it can optimises.</p> </article> <article> <h1>Distributed Computing for Health Data - 6 - Spark and Spark Components</h1> <p>Gonçalo Leiria — Sun, 18 Jan 2026 10:08:56 +0000</p> <p>These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.</p> <h3 id="lesson-6---spark-and-spark-components">Lesson 6 - Spark and Spark Components</h3> <p><strong>What is Spark</strong></p> <ul> <li>Software platform for data engineering, data science, and machine learninng</li> <li>Designed to be fast and general purpose</li> <li>Multi-language: Python, R, SQL, Java, Scala.</li> </ul> <hr> <p><strong>Spark vs Hadoop</strong></p> <p><a href="https://gleiria.github.io/blog/distributed-computing-course/distributed_computing_for_health_data_5/#dist-comp-rationale">See paragraph relevant section on lesson 5</a></p> </article> <article> <h1>Distributed Computing for Health Data - 5 - Hadoop Basic Usage</h1> <p>Gonçalo Leiria — Tue, 13 Jan 2026 07:25:55 +0000</p> <p>These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.</p> <h3 id="lesson-5---hadoop-basic-usage">Lesson 5 - Hadoop Basic Usage</h3> <p>In this lesson there is a lot of information on different options for installing Hadoop, how to navigate the GUI, and how to connect to a remote master node via SSH. Here, I’ll simply put down some notes that I think are important to retain and keep in mind. In the next lessons there will be more hands-on data manipulation using Spark and Hadoop.</p> </article> <article> <h1>Distributed Computing for Health Data - 4 - Hadoop</h1> <p>Gonçalo Leiria — Mon, 12 Jan 2026 06:41:44 +0000</p> <p>These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.</p> <h3 id="lesson-4---big-data-processing-tools-hadoop">Lesson 4 - Big Data Processing Tools: Hadoop</h3> <hr> <p><strong>Hadoop</strong></p> <ul> <li>Framework for distributed processing of large datasets</li> <li>Allows an application to scale up from a single computer to thousands of machines while implementing data distribution and fault tolerance</li> <li>Primarily intended for batch processing</li> <li>Hadoop is an ’ecosystem’ comprised of core and specific modules, each providing a different service</li> <li>At its core, Hadoop has a distributed file system and a distributed processing layer, HDFS and MapReduce</li> </ul> <p>Fault tolerance is the capability to continue operating smoothly despite failures or errors in one or more components of a distributed system. This resilience is crucial for maintaining system reliability, availability, and consistency.</p> </article> <article> <h1>Distributed Computing for Health Data - 3 - Distributed Computing</h1> <p>Gonçalo Leiria — Thu, 08 Jan 2026 14:05:28 +0000</p> <p>These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.</p> <h3 id="lesson-3---distributed-computing">Lesson 3 - Distributed Computing</h3> <hr> <p><strong>Distributed Computing</strong></p> <ul> <li>Use of autonomous networked machines (nodes) for solving a single problem</li> <li>Autonomous as each node only has access to its memory, processor and storage</li> <li>No shared memory across nodes so they communicate via messages and carrying data</li> <li>Data and processing tasks are distributed across nodes</li> </ul> <hr> <p><strong>Data Parallelism</strong></p> </article> <article> <h1>Distributed Computing for Health Data - 2 - Big Data Sources</h1> <p>Gonçalo Leiria — Tue, 06 Jan 2026 15:03:10 +0000</p> <p>These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.</p> <h3 id="lesson-2---big-data-sources">Lesson 2 - Big Data Sources</h3> <hr> <p><strong>Data Types and Data Sources</strong></p> <p>There are 5 data types:</p> <ul> <li>Structured (e.g. Electronic Health Records)</li> <li>Semi-structured ( e.g. EHR and vital-signs for a patien but not vital-signs for a second patient)</li> <li>Unstructured (CT scan)</li> <li>Time-series (ECG)</li> <li>Sequence data (RNA or DNA sequencing for example)</li> </ul> <hr> <p><strong>Structured Data:</strong></p> </article> <article> <h1>Distributed Computing for Health Data - 1 - Key Concepts</h1> <p>Gonçalo Leiria — Mon, 05 Jan 2026 13:31:45 +0000</p> <p>These are my personal notes related to the course Distributed Computing for Health Data provided by Health Data Research UK. I use this space to consolidate concepts and revisit material over time.</p> <h3 id="lesson-1---key-concepts">Lesson 1 - Key Concepts</h3> <hr> <p><strong>The 3 Vs of Big Data</strong></p> <p>A classic way to understand big data is through the 3 Vs:</p> <ul> <li><strong>Volume</strong> - how much data we have</li> <li><strong>Velocity</strong> - how fast data is generated and needs to be processed</li> <li><strong>Variety</strong> - how many different forms the data takes</li> </ul> <p>In healthcare, all three Vs show up at once and that’s what makes the domain both challenging and powerful.</p> </article> <article> <h1>Microsimulations - Digital Twins</h1> <p>Gonçalo Leiria — Sat, 01 Nov 2025 11:43:42 +0000</p> <p>The idea of using computers to simulate real-world systems has been around for many decades. However, in recent years, with the democratisation and significant cost reductions of computational power, increased data availability, and advancements in artificial inteligence, computational simulations have reached new levels of sophistication and their applications, particularly in the healthcare care domain, are expanding very rapidly. This blog post provides a very brief overview on microsimulations and how integrating machine learning and multi-objective optimisation into these systems can help us to tackle a concrete healthcare challenge: optimising type 1 diabetes screening strategies while minimizing costs associated.</p> </article> <article> <h1>Genetic_algorithms</h1> <p>Gonçalo Leiria — Mon, 03 Mar 2025 18:17:58 +0000</p> <p>Adding test here vdvdffdgflw;gjrelkw;gjrefklwghjnkrfngnfdlksn</p> </article> <article> <p>Gonçalo Leiria — Mon, 01 Jan 0001 00:00:00 +0000</p> <p>Please do get in touch if you have any questions or comments.</p> <p>Email: <a href="mailto:gleiria.dev@outlook.com">gleiria.dev@outlook.com</a></p> </article> <article> <p>Gonçalo Leiria — Mon, 01 Jan 0001 00:00:00 +0000</p> <p>Hi, I’m Gonçalo, welcome to my personal website 🚀.</p> <p>I’m a Data and Software Engineer at the University of Exeter in the United Kingdom. Currently, I’m working at the intersection of software engineering, the development of digital twins, and building Django web applications, with a particular focus on decision suport systems in health care. Prior to this, I have worked as a bioinformatician, mainly involded in projects using single-cell RNA sequencing data analysis to explore the underlying molecular mechanisms of health, development and disease. I am passionate about software engineering, data engineering, and continuous learning. I use this website as a place to document things I am learning and interested in.</p> </article> </main></body></html>