Inside an Engineer’s Biggest Technical Challenge — and Its Impact on Her Career

by Kelly O'Halloran
March 24, 2021
helix engineering san francisco
Ary Ascencio, Software Engineer at Helix

Upon realizing its legacy code was entangled, the genomics platform engineering team at Helix had a decision to make. 

Should they continue down the rabbit hole of debugging old code each and every time they added new features? Or should they rebuild the platform? 

“When hitting a wall, sometimes the solution is not to keep trying to break through it, but to walk around it,” said Engineer Ary Ascencio.
 

When hitting a wall, sometimes the solution is not to keep trying to break through it, but to walk around it.” 


Ultimately, the team launched a rebuild, setting in motion Ascencio’s biggest technical challenge in her career to date. While difficult and time consuming, the decision to build from scratch had lasting impacts on both Ascencio and the broader Helix engineering org

Below, Ascencio filled us in what led to her team’s decision — and the outcomes that followed.

 

More on Helix:SF Tech Leaders on How They’re Building Inclusive Brands

 

What’s the biggest technical challenge you’ve faced recently in your work? 

Ary Ascencio: An incremental feature scope snuck up on the team and turned what should have been a simple data aggregation task in a function run via AWS Lambda into a multi-headed beast.

We realized we were essentially working with legacy code. The codebase had passed through multiple teams, the Lambda’s usage had evolved over time. The different pieces of functionality had become entangled. We ran into domino effects whenever we attempted to add new features. At one point, seemingly non-deterministic behavior began consuming a lot of engineering time.

 

How did you and your team overcome this challenge? 

In the end, the team chose to freeze the existing codebase and rebuild from scratch. It became evident that refactoring the existing code would only lead to more complexity and result in inefficient use of engineering time.

We used production logs to quickly determine which features of the original codebase were in active use. From there, Confluence became a key tool for cross-team communication and documentation. It helped ensure the plans for the new Lambdas captured the nuances of the old one.

Once that was set, we leveraged our serverless infrastructure to generate the new AWS Lambdas while leaving the original code and Lambda intact. This meant we were able to run the new and old code in parallel as we rebuilt the cleaner, decomposed functionality of the legacy Lambda.

 

How did this experience help you grow as an engineer?

This challenge was a lesson in learning how to make the choice between repeatedly attempting to fix something and knowing when to start over. When hitting a wall, sometimes the solution is not to keep trying to break through it but to walk around it.

Additionally, the experience was a reminder of the importance of approachable documentation to reduce cognitive overload. In particular, it emphasized a need for creating standardized templates to reduce the burden of creating and updating docs. Getting decision-making guidance into written form became a useful tool for internal engineer planning, as well as communication with stakeholders. This improved documentation will prevent future codebases from becoming shrouded in mystery.

 

Jobs at Helix

San Francisco startup guides

LOCAL GUIDE
Top Software Engineer Jobs in San Francisco
LOCAL GUIDE
Best Companies to Work for in San Francisco
LOCAL GUIDE
Women in Tech: San Francisco Bay Area
LOCAL GUIDE
Best Sales Jobs in San Francisco Bay Area