Build an Internal Developer Platform with GitOps: Part 1

In the rapidly evolving landscape of platform engineering, our primary mission is to simplify and enhance the developer experience. This two-part series dives deep into the creation of an internal developer platform (IDP), showcasing how GitOps revolutionizes infrastructure management and application deployment. We utilize a combination of cutting-edge tools: Backstage, Crossplane, and ArgoCD. In Part 1, we focus on the platform engineer persona, demonstrating the automation of Google Kubernetes Engine (GKE) clusters to set the stage for seamless developer operations.

Video

Video Chapters

  • 00:00 Introduction

  • 00:42 Diagram and Explanation

  • 02:07 Backstage and Crossplane Courses Announcement

  • 02:33 Demo Starts

  • 13:06 GKE Cluster Backstage Template

  • 16:10 GitHub Actions Workflow

  • 18:49 Closing Remarks

Code

Join the Newsletter and get FREE access to the Source Code

The Essence of GitOps in Platform Engineering

GitOps, a methodology gaining momentum within software engineering organizations, serves as the practice of using Git repositories as the source of truth for defining and managing the state of infrastructure and applications. But what does a platform engineer do exactly with GitOps? They leverage it to streamline the creation, maintenance, and scaling of infrastructure, ensuring a consistent and reproducible environment that fosters innovation and agility for engineering teams.

Leveraging Backstage for a Unified Developer Portal

The journey begins with Backstage, a robust developer portal that orchestrates the entire process. It not only enhances the developer experience by providing a single pane of glass for all operational needs but also integrates seamlessly with other internal platform engineering tools, making it a cornerstone of our internal developer platform (IDP).

Crossplane: Declarative Infrastructure as Code

Crossplane extends Kubernetes to model complex infrastructures as simple, declarative configurations. In our journey, Crossplane plays a pivotal role in managing GKE resources, allowing us to define our cluster and node pool configurations as code, which can then be versioned, tracked, and managed through Git workflows. This exemplifies what platform engineering entails, particularly in managing underlying technologies through code.

Continuous Delivery with ArgoCD

ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, ensures that the state of our GKE clusters matches the configurations defined in our Git repositories. It automates the deployment process, enabling us to maintain a high velocity of change while ensuring stability and reliability, embodying the collaboration between development and operations teams.

The Workflow: From Concept to Cluster

Within our internal developer portal, Backstage is the starting point, where we select a GKE cluster template that triggers a GitHub Actions workflow. This workflow, in turn, interacts with ArgoCD to deploy a Crossplane claim within our Kubernetes infrastructure, ultimately resulting in the creation of a GKE cluster. This seamless, code-driven approach exemplifies the power of GitOps in automating infrastructure provisioning, a testament to successful platform teams.

Demo: A Closer Look at Building Internal Developer Platforms

Demo Diagram

Our demo showcases the entire process from the platform engineering team’s perspective, emphasizing the role of GitHub Actions in bridging Backstage with ArgoCD and Crossplane. This provides a clear, executable example of cloud platform engineering in action. Internal developer portals serve as a hub for this activity, enabling developer self-service and self-service capabilities, reducing the cognitive load on developers, and allowing them to focus on their core development process. Below I’ll only touch on the general steps, for more details, please watch the video above.

1. Backstage Template Parameters

The first step starts with Backstage as our internal developer portal. A senior platform engineer would create a template with cluster parameters. Other platform engineers would fill out these parameters to create the GKE cluster. It’s up to the platform engineering team whether to make this template available via RBAC to the development teams or keep it accessible only for the platform engineering team.

Backstage Template Parameters

Once you submit this template, a new GitHub repo gets created with the Crossplane configuration. Moreover, a GitHub actions workflow registers an ArgoCD application.

ArgoCD Application Created

We can now see that the application that will deploy our GKE cluster using Crossplane is created in ArgoCD. Give this about 15 minutes for Crossplane to successfully create the GKE cluster.

ArgoCD App Created

GKE Cluster Created

In our GCP console, we can see that our GKE cluster is finally created and ready to go.

GKE Cluster Created

Registering our GKE Cluster in ArgoCD

We need to register our cluster in ArgoCD to be used by our developers for their applications. Below is what it looks like in ArgoCD’s UI. Don’t worry about the connection status saying unknown at this point. It will turn to “Successful” later.

Registering our GKE Cluster in ArgoCD

Adding the New GO API Template in Backstage

Finally, we add a new GO API template in Backstage to be used by our Development teams later on in Part 2 of this series. Notice that it’s the platform engineers’ responsibility to create this template for the developers. Our job as platform engineers is done. We have enabled developer self-service capabilities.

Adding the New GO API Template in Backstage

Platform Engineering Examples: The Foundation of an IDP

This demonstration serves as a practical example of building the foundational layer of an internal developer platform. It highlights how the internal developer platform capabilities that include tools such as Backstage, Crossplane, and ArgoCD can be orchestrated to automate the provisioning of cloud infrastructure, paving the way for a more dynamic and efficient development lifecycle.

Frequently Asked Questions

How do you build a platform engineering team?

Building platform teams involves several key steps:

  1. Identify the Goals: Understand what you want to achieve with your platform. This could range from improving developer productivity and enhancing deployment workflows to creating a more scalable infrastructure.
  2. Assess Skills Needed: Determine the mix of skills required, such as expertise in cloud services, infrastructure as code, CI/CD practices, container orchestration (e.g., Kubernetes), and familiarity with platform engineering tools (e.g., Backstage, Crossplane, ArgoCD).
  3. Hire for Culture and Skill: Look for individuals who not only have the technical skills but also fit well with the team culture. Emphasize the importance of collaboration, continuous learning, and the willingness to embrace new technologies.
  4. Promote Knowledge Sharing: Encourage team members to share their expertise and learn from each other, fostering a culture of continuous improvement.
  5. Invest in Training: Provide opportunities for professional development to keep the team updated with the latest trends and technologies in platform engineering.

What does a platform engineer do?

A platform engineer focuses on creating and managing a shared platform that developers use to build and deploy applications. Responsibilities include:

  • Automating infrastructure provisioning and management using infrastructure as code.
  • Implementing CI/CD pipelines for automated testing and deployment.
  • Ensuring the platform’s reliability, scalability, and security.
  • Providing tools and services that enhance developer productivity.
  • Continuously optimizing the platform based on feedback and evolving requirements.

What does a platform engineering team do?

A platform engineering team is responsible for building and maintaining the underlying platforms that support software development and operations within an organization. Their primary goal is to enhance developer productivity and operational efficiency through automation, tooling, and best practices. Key responsibilities include:

  • Developing Internal Developer Platforms (IDPs): Creating centralized platforms that offer developers the tools, services, and APIs they need to build, test, deploy, and monitor their applications efficiently.
  • Implementing Infrastructure as Code (IaC): Utilizing code to manage and provision infrastructure, ensuring environments are reproducible, scalable, and manageable.
  • Establishing CI/CD Pipelines: Setting up continuous integration and continuous deployment pipelines to automate the software release process, from code commit to production deployment.
  • Ensuring Reliability and Scalability: Designing the platform for high availability, scalability, and fault tolerance to support dynamic workloads and minimize downtime.
  • Maintaining Security and Compliance: Implementing security best practices and policies to protect applications and data, and ensuring compliance with relevant regulations.
  • Facilitating Collaboration and Self-Service: Promoting a culture of collaboration between development, operations, and business teams, and empowering developers through self-service capabilities.

Do I need a platform team?

Whether you need a platform team depends on several factors:

  • Scale and Complexity: Larger organizations or those with complex deployments may benefit significantly from a dedicated platform team.
  • Development Velocity: If speeding up development cycles and improving CI/CD workflows are priorities, a platform team can provide the necessary infrastructure and tooling.
  • Operational Efficiency: A platform team can help standardize development practices and reduce operational overhead, making it easier to manage and scale applications.

What is an open developer platform?

An open developer platform is a framework or environment that is openly available for developers to build, deploy, and manage applications. These platforms often offer APIs, SDKs, tools, and services that developers can use to create software solutions. Open developer platforms are designed to be extensible, customizable, and accessible, fostering innovation and collaboration within the developer community.

Closing Thoughts and What’s Next

As we wrap up Part 1, we’ve laid the groundwork for our IDP, demonstrating how GitOps principles can be applied to automate infrastructure management effectively. In Part 2, we’ll switch perspectives to the developer persona, exploring how to deploy a GO API application using this robust infrastructure. Stay tuned as we continue to explore the intricacies of building an Internal Developer Platform with GitOps, bridging the gap between infrastructure and application development for a seamless developer experience.

Suggested Readings

Scroll to Top