During the weekend of April 18-19, I sought out and participated in a data engineering hackathon (Mage Battleground 2023) with my friend Mattan Tseng!
The theme of this hackathon is “Craft insights from real-time customer behavior analysis.”
This hackathon was also a great opportunity for me to get out of my comfort zone, as well as, learn about data engineering and creating data pipelines.
What I learned:
- Docker – Docker Desktop is pretty cool and the entire concept of containerization is neat. I used Windows Powershell to install Mage through Docker, and I did most of my work through Docker
- Git command line – I’ve used it in the past on several projects, but I always appreciate a refresher and an opportunity to use Git command line.
- Presentation – This was my first time being really hands-on with data engineering, so I wanted the data to drive the presentation. We were able to get some pretty neat graphs from Mage.
- Data Engineering Pipelines – This was totally new to Mattan and me, so we spent a good chunk of time Googling what data engineering pipelines consisted of. We created a high-level diagram of our data pipeline that can be seen in the presentation.
Next steps for me:
- We ran into issues with the Git integration with Mage, so this could be something to look into further
- Data engineering is actually really neat!
- I can’t wait to learn more about it as ODSC (where I heard about this hackathon) hosts and announces more events
Roselynn Conrady 