A Korean proverb many Koreans like myself often forget is:
옷깃만 스쳐도 인연
Merely brushing by someone means you’re fatefully connected to that person.
So many social connections come and go through our lives, and we don’t always value them as much as they deserve. But so much can happen even with brief interactions, which is what I’m going to unfold in this post.
In 2014, I was having busy days as a lead of the Netty project. Like any moderately popular open-source project, its maintenance is often a race against your inbox. Overwhelmed by an abundance of questions and feature requests, it’s easy to focus too much on achieving ‘inbox zero’ and forget about the milestones users are trying to achieve with the project.
Apache Spark™ was one such user, and that was my first brush with Reynold Xin, Databricks’ Co-founder. Norman (another Netty maintainer) and I helped him with a few interesting issues related to zero-copy file transfer and /dev/epoll transport. We didn’t even know what project Reynold was working on at that time. A few months later, we finally learned from Reynold’s email about Apache Spark’s winning at 2014 Gray Sort competition that Netty played a non-trivial role. It was a very pleasant surprise, and I was excited that Netty could make an impact in many different areas.
Exciting moments are fleeting and the life of an open-source project maintainer continues. We, maybe Reynold, too, all forgot about this milestone and moved on to the next pile of issues in an issue tracker.
Fast forward — my focus moved from pure network programming to frameworks that help organizations scale by providing smooth migration paths to asynchronous and reactive programming models. Armeria was the open-source RPC/REST framework my team designed from scratch for this purpose. In the last few years, thanks to Armeria, I had the opportunity to help users from various companies who were facing similar migration challenges.
In its essence, the journey with Armeria was not largely different from what I had with Netty. I was thrilled by our happy users and the pleasant surprises they gave us. At the same time, it was a race against an endless stream of issues and pull requests to address and review. We shipped more than 170 releases with 2,500 commits for 5 years. It was a great achievement, but it also made me crave something new to work on that would broaden my horizon.
That was when my and Databricks’ paths crossed again.
Earlier this year, the Databricks engineering team was designing a new communication protocol for the recently announced SQL Analytics product. As part of the researching solutions for a scalability issue, they found Armeria. The team reached out to me explaining the challenges they had with sending massive amounts of data across the network, with the goal to optimize for both higher throughput and lower latency. A series of interesting technical discussions led to Reynold asking whether I’d consider joining Databricks.
I must confess that, after all those years, I had no idea what Databricks does! Actually, I couldn’t even remember the name of the company. After brief research about the company, I was even less sure whether Databricks would be the right place for me given my lack of experience in data or machine learning. I decided to talk to the team more since the technical interactions were interesting (sorting petabytes of data and protocol optimizations).
However, my impression of the company changed dramatically as I went through the interview process. The interviews were more like bidirectional technical discussions, even when I was solving a given problem. I was impressed that I was not treated as a student to fill in the blanks but as a partner with the same goal. All in all, the interviewers were very friendly, and I was able to express my ideas interactively with great comfort. Such an attitude does not come from a recruiting manual. It was a clear sign that Databricks is a healthy place built on top of good faith and curiosity.
What fascinated me was the engineers’ telltale excitement at every moment: when explaining what they’ve built, showing some growth charts, and confessing the challenges ahead–or even some technical debt. At the end of the interview day, I started to think Databricks may be a place worth betting my career on.
As you can guess, the rest is history – well, in progress. As part of our engineering team, I focus on our RPC stack to help scale the organization and optimize our software fleet of millions of machines. The three short months here reinforce the choice I made. I love the technical challenges, in-depth design review process, investment in new technologies, and heavy leverage of automation I found here. I also enjoy the impact I have had already, and will have, on the future, with the great culture and learnings from the rest of the team.
Looking back, I’m amazed at how such a brief cross-continental collaboration led us here – even after many years. It is indeed a magical journey that leaves a lot to be discovered and remembered.
(This post originally appeared at Databricks company blog.)