2nd Place at Data Engineering Hackathon ๐Ÿง™๐Ÿช„

During the weekend of April 18-19, I sought out and participated in a data engineering hackathon (Mage Battleground 2023) with my friend Mattan Tseng!

The theme of this hackathon is “Craft insights from real-time customer behavior analysis.”

This hackathon was also a great opportunity for me to get out of my comfort zone, as well as, learn about data engineering and creating data pipelines.

What I learned:

  • Docker – Docker Desktop is pretty cool and the entire concept of containerization is neat. I used Windows Powershell to install Mage through Docker, and I did most of my work through Docker
  • Git command line – I’ve used it in the past on several projects, but I always appreciate a refresher and an opportunity to use Git command line.
  • Presentation – This was my first time being really hands-on with data engineering, so I wanted the data to drive the presentation. We were able to get some pretty neat graphs from Mage.
  • Data Engineering Pipelines – This was totally new to Mattan and me, so we spent a good chunk of time Googling what data engineering pipelines consisted of. We created a high-level diagram of our data pipeline that can be seen in the presentation.

Next steps for me:

  • We ran into issues with the Git integration with Mage, so this could be something to look into further
  • Data engineering is actually really neat!
    • I can’t wait to learn more about it as ODSC (where I heard about this hackathon) hosts and announces more events

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s