“In science if you know what you are doing you should not be doing it. In engineering if you do not know what you are doing you should not be doing it.”

― Richard Hamming, The Art of Doing Science and Engineering

Just returned from Padova after attending SIGIR 2025, the most prestigious conference on Information Retrieval (IR) in the world (A* CORE Ranking). An experience full of learnings - but I was struck by the amount of research that could be applied in the industry already. It’s evident that the gap between research and industry is incredibly narrow nowadays.

A bit of context: Where LLMs fall short on things like actual and truthful answers - two essential factors for the legal industry -, IR capabilities come to the rescue. The rapid pace of AI development (especially in IR) demands a tight integration between research and product development. In other words, experimentation must be embedded into product development, not treated as an afterthought, a nuisance or a nice-to-have.

Only experimentation will determine whether something from academia is applicable, but also if something is hype or not. This means that research and product development need to walk hand-in-hand, between science and engineering, between experimentation and implementation, between uncertainty and the familiar. And companies need to take shots at delivering the best of what AI has to offer to customers. At the end of the day, customers are the ones who matter the most.

And the dance between science and engineering is a very hard dance to dance; but far from impossible.

Traditional approaches handle experiments “when there’s time” and some claim “it takes too long”, not understanding that they’re creating AI legacy by doing that. AI development is far more fluid and experimental than traditional development. Unknowns happen way more often and some scramble for safety - which comes in the form of confidence. However, confidence is still being mistaken for competence.

There’s also a trade-off here (hence being a dance): too much experimentation and there’s a risk of low rate of productionalization, whereas too much engineering and there’s a risk of AI legacy. The companies who release products in tune with this balance between science and engineering will come out on top.

But for now, in my role, I have to find what AI capabilities are there that can benefit our customers. And SIGIR was the perfect place for that. There’s a lot to unpack in this topic which I’ll keep writing about. Feel free to follow me here.

Thanks to Karnov for enabling this learning opportunity at SIGIR 2025 in Padova.

The complete code is available here.

Intro

Before Task Groups in Airflow 2.0, Subdags were the go-to API to group tasks. With Airflow 2.0, SubDags are being relegated and now replaced with the Task Group feature. The TaskFlow API is simple and allows for a proper code structure, favoring a clear separation of concerns.

What we’re building today is a simple DAG with two groups of tasks, using the @taskgroup decorator from the TaskFlow API from Airflow 2. The graph view is:

What this pipeline does is different manipulations to a given initial value. The init() task instantiates a variable with the value 0. It then passes to a group of subtasks (group_1) that manipulate that initial value. group_2 will aggregate all the values into one. Finally, the end() subtask will print out the final result.

Let’s get started by breaking the pipeline down into parts.

Grouping tasks - breakdown

As you can see in the image above, there’s an init() and end() task. In between, there are two groups of tasks, but let’s start with the first and last task of the pipeline.

init() task and end() task

The init() task is the starting point for this pipeline - it returns the initial value that will be manipulated throughout the pipeline: 0. The end() task will print out all the manipulations in the pipeline, to the console.

Let’s look at the code for the init() task:

@task
def init():
  return 0

That’s it. Now the code for the end() task:

@task
def end(value):
    print(f'this is the end: {value}')

It’s also quite simple to define the flow of the whole pipeline, returned by the function that wraps everything:

return end(group_2(group_1(init())))

Looking at the code above it’s possible to see that:

1. The init() function “feeds” group_1;
2. The result of group_1 is “sent” to group_2;
3. end() receives the outcome of group_2.

Task Group #1 (group_1)

group_1 has a set of three tasks that manipulate the original number:

# This task group has three subtasks:
# each subtask will perform an operation on the initial value
# this group will return a list with all the values of the subtasks
@task_group(group_id='group_1')
def group_1(value):

  # The @tasks below can be defined outside function `group_1`
  # What matters is where they are referenced
  @task(task_id='subtask_1')
  def task_1(value):
    task_1_result = value + 1
    return task_1_result

  @task(task_id='subtask_2')
  def task_2(value):
    task_2_result = value + 2
    return task_2_result

  @task(task_id='subtask_3')
  def task_3(value):
    task_3_result = value + 3
    return task_3_result

  # tasks are referenced here
  task_1_result = task_1(value)
  task_2_result = task_2(value)
  task_3_result = task_3(value)

# sending this list to `group_2`
return [task_1_result, task_2_result, task_3_result]

The group_1 function receives the result from the init() task. And notice what’s being returned here: a list of the three values. Each of the value stems from subtask_1, subtask_2 and subtask_3. This list of values is what’s going to be sent to group_2.

Task Group #2 (group_2)

group_2 is rather simple. It receives the list sent from group_1 and sums all values (subtask_4 does it) and then subtask_5 just multiplies by two the result from task_4:

@task_group(group_id='group_2')
def group_2(list):

  @task(task_id='subtask_4')
  def task_4(values):
    return sum(values)

  @task(task_id='subtask_5')
  def task_5(value):
    return value*2

  # task_4 will sum the values of the list sent by group_1
  # task_5 will multiply it by two.
  task_5_result = task_5(task_4(list))

  return task_5_result

That’s it - the next task is the end(), and it has been handled before in this post. Again, it’s possible to see the full code here.

If you check the log of the end() task (see my previous post to know how to check for task logs), you’ll see the result printed. The final result should be 12.

Success! 🎉

Conclusion

Creating task groups in Airflow 2 is easy - it removes complexity that existed before and allows creating pipelines with clean code.

Background

This blog post is part of a series where an entire ETL pipeline is built using Airflow 2.0’s newest syntax and Raspberry Pis. It entails knowledge of some terms, so here’s a great place to refresh memory. Also, check my previous post on how to install Airflow 2 on a Raspberry Pi.

The full code is on Github.

Create a DAG definition file

We’ll start by creating a DAG definition file inside the airflow/dags folder:

touch example.py

Let’s populate it by adding a DAG.

DAG breakdown

The @dag decorator

A DAG has tasks. In this example, it has two tasks where one is dependent on the result of the other. For this, we’ll be using the newest airflow decorators: @dag and @task.

We start by defining the DAG and its parameters. We’ll determine the interval in which the set of tasks should run (schedule_interval) and the start date (start_date). Of course, there are other parameters to chose from, but we’ll keep the scope to the minimum here.

# Use the DAG decorator from Airflow
## `schedule_interval='@daily` means the DAG will run every day at midnight.
## It's possible to set the schedule_interval to None (without quotes).
@dag(schedule_interval='@daily', start_date=days_ago(2))
# The function name will be the ID of the DAG.
# In this case, it's called `EXAMPLE_simple`.
# In this case it's called `EXAMPLE_simple`.
def EXAMPLE_simple():


# Placeholder for the tasks inside the DAG
# ...

# run the dag
dag = EXAMPLE_simple()

Notice the @dag decorator on top of the function EXAMPLE_simple. The function name will also be the DAG id. In the end, we just run the function of the DAG.

The @task decorator

Now that the @dag wrapper is settled, we need to define the two tasks inside. Remember, this DAG has two tasks: task_1 generates a random number and task_2 receives the result of the first task and prints it, like the following:

# ...
    @task
    def task_1():
        return random()

    @task
    def task_2(value):
        print(f'The randomly generated number is {value} .')

    # This will determine the direction of the tasks.
    # As you can see, task_2 runs after task_1 is done.
    # Task_2 then uses the result from task_1.
    return task_2(task_1())
# ...

Visually, the DAG graph view will look like this:

The code before and after refers to the @dag operator and the dependencies. Next, we’ll put everything together:

from airflow.decorators import dag, task
from airflow.utils.dates import days_ago

from random import random

# Use the DAG decorator from Airflow
# `schedule_interval='@daily` means the DAG will run everyday at midnight.
# It's possible to set the schedule_interval to None (without quotes).
@dag(schedule_interval=None, start_date=days_ago(2), catchup=False)
# The function name will be the ID of the DAG.
# In this case it's called `EXAMPLE_simple`.
def EXAMPLE_simple():

    @task
    def task_1():
        # Generate a random number
        return random()

    @task
    def task_2(value):
        # Print the random number to the logs
        print(f'The randomly generated number is {value} .')
        print(f'The randomly generated number is {value}.')

    # This will determine the direction of the tasks.
    # As you can see, task_2 runs after task_1 is done.
    # Task_2 then uses the result from task_1.
    return task_2(task_1())


dag = EXAMPLE_simple()

That’s it. Let’s run this.

Running the DAG

Once the DAG definition file is created, and inside the airflow/dags folder, it should appear in the list. Now we need to unpause the DAG and trigger it if we want to run it right away. There are two options to unpause and trigger the DAG: we can use Airflow webserver’s UI or the terminal. Let’s handle both.

Run via UI

First, you should see the DAG on the list:

In this example, I’ve run the DAG before (hence some columns already have values), but you should have a clean slate.

Now we enable the DAG (1) and trigger it (2), so it can run right away:

Click the DAG ID (in this case, called EXAMPLE_simple), and you’ll see the Tree View. Having triggered a new run, you’ll see that the DAG is running:

Heading over to the Graph View, we can see that both tasks ran successfully 🎉:

But what about the printed output of task_2, which shows a randomly generated number? We can check that in the logs.

Checking the logs via UI

Inside Graph View, click on task_2, and click Log.

It’s possible to see the output of the task:

Success again 🎉!

Run via terminal

An alternative to the UI, when it comes to unpause and trigger and DAG, is straightforward. Knowing the ID of the DAG, then all we need is: An alternative to the UI, when it comes to unpause and trigger a DAG, is straightforward. Knowing the ID of the DAG, then all we need is:

airflow dags unpause EXAMPLE_simple && airflow dags trigger EXAMPLE_simple

Checking the logs via terminal

Assuming your airflow installation is in the $HOME directory, it’s possible to check the logs by doing:

cd ~/airflow/logs/EXAMPLE_simple/task_2

And select the correct timestamp (in my case it was):

tail 2021-08-06T14:57:35.762094+00:00/1.log

Where the output should include:

Followed by the actual number we’ve generated in this run.

Conclusion

This is a beginner’s friendly DAG, using the new Taskflow API in Airflow 2.0. It’s possible to create a simple DAG without too much code. In the next post of the series, we’ll create parallel tasks using the @task_group decorator.

This is the first post of a series, where we’ll build an entire Data Engineering pipeline. To follow this series, just subscribe to the newsletter.

Install dependencies

Let’s make sure our OS is up-to-date.

sudo apt-get update -y
sudo apt-get upgrade -y

Now, we’ll install Python 3.x and Pip on the Raspberry Pi.

sudo apt-get install python3 python3-pip

Airflow relies on numpy, which has its own dependencies. We’ll address that by installing the necessary dependencies:

sudo apt-get install python-dev libatlas-base-dev

We also need to ensure Airflow installs using Python3 and Pip3, so we’ll set an alias for both. To do this, edit the ~/.bashrc by adding:

alias python=$(which python3)
alias pip=pip3

Alternatively, you can install using pip3 directly. For this tutorial, we’ll assume aliases are in use.

Install Airflow

Create folders

We need a placeholder to install Airflow.

cd ~/
mkdir airflow

Install Airflow package

Finally, we can install Airflow safely. We start by defining the airflow and python versions to have the correct constraint URL. The constraint URL ensures that we’re installing the correct airflow version for the correct python version.

# set airflow version
AIRFLOW_VERSION=2.1.2

# determine the correct python version
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"

# build the constraint URL
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

# install airflow
pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

Initialize database

Before running Airflow, we need to initialize the database. There are several different options for this setup: 1) running Airflow against a separate database and 2) running a simple SQLite database. The SQLite database is in use in this tutorial, so there’s not much to do other than initializing the database.

So let’s initialize it:

airflow db init

Run Airflow

It’s now possible to run both the server and the scheduler:

airflow webserver -p 8080 & airflow scheduler

Now open http://localhost:8080 on a browser. If you need to log in, you’ll need to create a new user. Here’s an example:

airflow users create \
    --username admin \
    --firstname Peter \
    --lastname Parker \
    --role Admin \
    --email spiderman@superhero.org

Once authenticated, it’s now possible to see the main screen:

And that’s it - you’ve now installed Airflow! Optionally, you can take extra steps.

Optional

Start airflow automatically

In order to start both the webserver and the scheduler automatically on system boot, we’ll need three files: airflow-webserver.service, airflow-scheduler.service, and an environment file. Let’s break this into parts:

1. Go to Airflow’s github repo and download the airflow-webserver.service and the airflow-scheduler.service

2. Paste them on the /etc/systemd/system folder.

3. Edit both files. Firstly, airflow-webserver.service should look like:

[Unit]
Description=Airflow webserver daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service

[Service]
EnvironmentFile=/home/pi/airflow/env
User=pi
Group=pi
Type=simple
ExecStart=/bin/bash -c 'airflow webserver --pid /home/pi/airflow/webserver.pid'
Restart=on-failure
RestartSec=5s
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Now moving on to edit airflow-scheduler.service file, which should look like:

[Unit]
Description=Airflow scheduler daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service

[Service]
EnvironmentFile=/home/pi/airflow/env
User=pi
Group=pi
Type=simple
ExecStart=/bin/bash -c 'airflow scheduler'
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

Notice that the user and Group have changed, as well as the ExecStart. You’ll also notice that there’s an EnvironmentFile that hasn’t been created yet. That’s what we’ll do now.

4. Create an environment file. You can call it any name. I chose to call it env and placed it on the /home/pi/airflow folder. In other words:

cd ~/airflow
touch env

Edit the env file and place the contents:

AIRFLOW_CONFIG=/home/pi/airflow/airflow.cfg
AIRFLOW_HOME=/home/pi/airflow/

5. Lastly, let’s reload the system daemons:

sudo systemctl daemon-reload
sudo systemctl enable airflow-webserver.service
sudo systemctl enable airflow-scheduler.service
sudo systemctl start airflow-webserver.service
sudo systemctl start airflow-scheduler.service

That’s it! What’s next?

In the next blog post of this Data Engineering series, we’ll create our first Directed Acyclic Graph (DAG) using Airflow. Subscribe to the newsletter, and don’t miss out!

Sources

1. https://airflow.apache.org/docs/apache-airflow/stable/installation.html
2. https://medium.com/the-kickstarter/apache-airflow-running-on-a-raspberry-pi-2e061f6c3655
3. http://www.thecrustyengineer.com/home/post/setting_up_airflow_on_a_raspberry_pi_4_part_1

Heart Rate Variability (HRV) is a way to measure the variation in time between each heartbeat. This variation is a measure of how the heart reacts to physical exercise, mental stress, and heart diseases, directly linked to an increased risk of mortality.

It has its origin in neurons from the parasympathetic, sympathetic nervous system, and vagus nerve. Evidence suggests that HRV is impacted by stress, specifically due to higher levels of stress resulting in a lower HRV.

While stress (and its causes and effects) is a known research topic, it’s also more accessible due to the widespread usage of wearables that allow the collection of HRV data. The combination of the possibility of stress analysis from HRV and easy access to data makes this the main focus of the present report, determining whether machine learning techniques can help minimalizing generalization errors.

This report is structured into three main parts: data analysis, supervised, and unsupervised learning. All three parts revolve around predicting and/or clustering HRV values.

Report

Download full report (41 pages)

Code

1. Data Preparation + Principal Component Analysis
2. Supervised learning - classification (baseline, logistic regression, neural network)
3. Supervised learning - regression (linear regression, neural network)
4. Unsupervised learning - Agglomerated Hierarquical Clustering, Gaussian Mixture Model, Anomaly Detection, Aprior Association