Joey Koh

Recent Notes

See 36 more →

Steel Thread Approach

7 min read

I elaborate on Steel Threading – a technical software engineering approach that can effectively navigate ambiguity

  • in building a new product or feature,
  • and especially in collaboration within unfamiliar teams.

Context

In over 40 hackathons I’ve done, I’ve built a lot of stuff.

  • From my context, often teaming with strangers I never met or worked with before
  • And usually, as team lead – with technical and non-technical leadership

I tested and found applying Steel Threading to be the most effective in technical leadership in the context of a hackathon.

I learnt about it from working as a Data Scientist/Data Engineer/SWE, when a Staff Engineer in my team was advising me to try it out when I was implementing a new encryption service for our data pipeline.

Definition

Essentially, Steel Threading is:

  • To as early as possible, build ONLY the basic components that would tie all separate layers of your product/feature together
  • To integrate early as simple and basic as possible
  • Imagine a Steel Thread piercing through your entire product tech stack and features,
  • Or a slice that would link everything together.
  • This slice can be a single simple use-case flow for the product, or even more fundamentally first – a slice purely for integration before that.
  • Answers the question, “What is the least that needs to be built and done to link everything together end-to-end?”
  • Can involve early deployment and I highly encourage it
  • The simpler the Steel Thread is, to cut through the noise, nice-to-haves, and non-essentials – the better.

A great way to think about it is,

What are the things we need to do regardless of what features we build? What’s the MVP of the MVP?

Well, for a webapp, you would need some sort of frontend, backend, and deployment to a live site done anyway – why not do it now rather than later?

Your team's separate pieces of work needs to be integrate regardless!

  • Rather than wait for the monster that is your codebase to grow large, complex and unruly.
  • Why not tie it all together while it’s still small and manageable?
  • Trying to integrate in the late game, can kill a team’s ability to meet the deadline.
  • And even if successful, would cost a lot more resources – both in output, energy, time and in spirit.

When to use?

Shines especially when,

  • New or unfamiliar team working together for the first time
  • New product or feature
  • Using new technology
  • Adding new components to old codebase
  • Ambiguity in product/use-cases/tech-stack

Less penalising to not use when,

  • Familiar team with experience working together
    • everyone knows what they need to do and who does what
    • normally, people would find some Steel Threading occurring anyway from this
  • Repeat of x from past work

Less relevant to use when,

  • Changes for existing product/feature

Examples

Roughly speaking, here are examples:

Steel Threaded - sparks joy

To build a typical webapp,

  1. Draw basic design diagram (super high-level)
  2. Pick tech stack and assign scopes of work
  3. Basic setup
    1. Git repository made and shared, proper version control established
    2. Concurrently, each team member does a subset:
      1. Backend framework spun up - can boilerplate/template it
      2. Single basic API call defined in Backend for Frontend (zero or perhaps single simple use-case functionality)
      3. Frontend setup - zero design and can possibly support that single simple use-case
      4. Setup automated CI/CD with GitHub actions (pretty easy to do nowadays)
      5. Setup hosting
  4. Integrate immediately
    1. Frontend calls single Backend defined API successfully
    2. Test local build for success
    3. Test automated CI/CD
    4. Test hosting
  5. Get live site that looks terrible and does absolutely nothing (or that single use-case) 🔥
  6. Ensure
    1. everyone knows how to contribute and test
    2. individuals know their integration points between each other

Not Steel Threaded - does not spark joy

To build a typical webapp, BOLDED are some significant differences

  1. Draw entire design diagram (high-level + low-level components)
  2. Pick and optimise tech stack, and assign scopes of work
  3. Basic setup
    1. Git repository made and shared, proper version control established
  4. Isolated development, each team member focuses on their subset:
    1. Backend built until core functionality achieved. Go get external APIs
    2. All API calls defined in Backend for Frontend (core functionality)
    3. Frontend built fully with user inputs/outputs flow
    4. Frontend iterations - with design input from Designer
    5. Stay on local
  5. Integrate intermittently
    1. Frontend calls multiple Backend defined APIs and throughout the sprint
    2. Figure out how to contribute work and integrate
    3. Tweak design and architecture
  6. Realise it’s 2 hours to the deadline
  7. Rush
  8. Integrate and test late
    1. Test consolidated local build for success
    2. Setup CI/CD pipeline and test
    3. Setup and test hosting
  9. Maybe, get live site that looks good and does some things 😎
  10. Realise everyone is stressed 🫠

Steel Thread Benefits

  • End-to-end implicit testing:

    • so if something breaks on the live site, you know its because of you and can fix early.

  • Immediate Feedback Pipeline:

    • Your team has a literal live product immediately
    • Changes are immediately reflected and that allows immediate feedback in development.
      • e.g.
      • Oh, that button looks weird with that menu
      • Ah, the user login doesn’t work
    • Plus, at an early point, you can launch it to a small group of users for early feedback!
      • e.g.
      • Ah, the user can generate profane images – we better guardrail that.

  • Modular: Every aspect is somewhat isolated/modular once Steel Thread is established

    • allows any developer to establish their own small world of sanity
    • that’s powerful - because you need to move fast and having agency and boundaries helps

  • Alignment:

    • Without significant time investment, it makes it super easy to relate a design diagram and the codebase from the get-go
    • Any changes can be easy made to the design diagram as work gets done and discovery occurs
      • = Maintaining alignment consistently

  • Extendable:

    • Within, not only are the modules freely changeable in that component of ownership.
    • But also, at the ends of each module.
    • So let’s say I wanted to improve the frontend
      • I could add more features first in my own layer
      • Then, because the Backend and I have established and tested that simple API route between us.
      • We both know that track between the mountain shack to the city, and can make more tracks easily.
      • I can relate that back to the Backend with confidence and technical conciseness.
    • There is an implicit path already established to add a new feature into the codebase.

  • Confidence: and Happiness

    • Every single person on the team knows that we have already made our path from the mountain shack, to the city, to the rural outback, and to Ayer’s rock.
    • We know we won’t get lost.
    • We know who is responsible for what and how to make more similar paths as needed.
    • We know how to update the map (design diagram)

  • Scalable: in the sense of increasing complexity/systems as needed:

    • E.g. user auth, payments, cloud, security…etc
    • Easy to add rooms and windows to your home when the foundation and walls are already there.
    • It provides focus on the core first – great for MVP’ing

  • Defined Boundaries: establishes where something ends, and where something begins

    • To avoid wastage of effort/duplication of work/integration complexity

    • When an engineering team starts building something

    • there is a process that occurs - to understand where and what people should be working on and responsible for

    • because, let’s say you have 2 frontend and 3 backend engineers - there will be some overlap

    • especially, if you have +1 AI engineer, ML engineer

    • especially, for a new team, that hasn’t worked together before

    • There is more uncertainty and unknowns

      • That will kill a team if not managed well

    • Now, speed is the significant priority in startups or hackathons

    • There is no time to go through this slowly and figure things out

    • Resource limitations: do not take kindly to duplication of work and effort wastage.

    • You need to establish clear boundaries (which may/may not be followed perfectly 😉)

    • Due to that, you need to as quickly as possible - make the steel thread and reach alignment

More info

I wrote this article straight from my brain and current reflections on my experience. It’s a good challenge to draw from my understanding.

So for a different take on it,
Check out https://www.rubick.com/steel-threads/ which I referred to when learning about Steel Threading years ago.

I’d love any comments and feedback to get your take and improve my own understanding.

Graph View

Backlinks