From Friction to Flow: Elevating Teamwork with Terraform, Vault, and Consul

Abstract

In this blog post, we’ll explore the common sources of friction that teams encounter when writing, deploying, supporting, and securing applications. We’ll examine various personas within a team and discuss how the HashiCorp stack, comprising Terraform, Vault, and Consul, can help alleviate these challenges and foster a high-functioning, content team.

Introduction

Friction is an ever-present challenge in the world of software development and IT operations. In today’s interconnected IT landscape, teams across various domains face a multitude of hurdles when it comes to writing, deploying, supporting, and securing applications and systems. This challenge transcends individual roles and responsibilities; it’s a collective issue that impacts everyone involved in delivering and maintaining modern applications.

In this blog post, we delve deep into the root causes of this friction, examining the challenges that teams encounter daily. We recognize the importance of selecting the right ecosystem of tools, tailored to meet the unique needs of different teams. Rather than focusing solely on specific solutions, our aim is to explore how organizations can choose a versatile set of tools that address specific pain points.

To address these challenges and create a smoother path forward, we’ll draw insights from real-world scenarios and best practices. By fostering collaboration and bridging the gap between various roles within an organization, this approach can transform friction points into opportunities for improved teamwork and productivity. Ultimately, it leads to happier customers and more efficient workflows in an ever-evolving IT landscape.

Meet the Personas

To understand the friction points better, let’s introduce some key personas within a typical cross-functional team involved in various aspects of software development, IT operations, and infrastructure management:

Jennifer - The Developer

• Jennifer is focused on writing code and delivering new features quickly.
• Her primary concern is not infrastructure or security but rather building applications efficiently.
• Frequent context switching disrupts her productivity.

Adi - The Platform Engineer

• Addie brings years of experience managing infrastructure and strives to create a robust platform.
• He faces challenges in managing multiple tools and coordinating between various teams.
• Constant firefighting and disruptions affect his focus and stability.

Hannah - The SRE (Site Reliability Engineer)

• Hannah aims to ensure system reliability and stability while seeking insights through metrics and KPIs.
• She seeks to shift from reactive to proactive operations.
• Inconsistent data and disjointed tools hamper her efforts.

Security Engineer - The Security Guru

• The security engineer’s role is ever-evolving, dealing with an overwhelming influx of security data.
• Their mission is to proactively address security vulnerabilities and establish a robust security posture from the beginning.
• Last-minute security adjustments in production can be challenging.

Identifying the Friction

The friction within a cross-functional team often arises from the conflicting priorities and demands of different personas. In some cases, there is a misinterpretation of expectations when adopting DevOps. It can be assumed that the intent is for all team members to become experts in the full stack, which creates a great deal of friction by detracting from an individual’s primary focus.

Developers may be required to become security experts, platform engineers struggle to manage a plethora of tools, SREs face inconsistent data, and security engineers grapple with a deluge of security data. To address these issues and the misaligned expectations, we must find a smoother path forward.

Building Purposeful Platforms

Building platforms with a clear purpose is crucial. Rather than adopting tools and technologies blindly, consider the specific needs of each persona and the shared goal of achieving happy customers. Here are some components that can be part of a purposeful platform:

• Development: Standard tools like GitHub and VS Code facilitate code development and collaboration.

• CI/CD: Implement a robust CI/CD pipeline for efficient application deployment.

• Infrastructure: Terraform offers a way to provision and manage infrastructure efficiently.

• Security: Vault ensures secure and dynamic credential management while promoting the principle of least privilege.

• Observability: Utilize industry-standard tools like Grafana and Prometheus to gain insights into system performance.

A Smoother Path Forward

Let’s explore how each persona can benefit from a smoother path forward enabled by Terraform, Vault, and Consul:

1. Jennifer - The Developer

   • Jennifer can focus solely on writing code without the need to become an expert in Vault or other security tools.
   • Vault provides dynamic credentials seamlessly, allowing her to access secure resources effortlessly.
   • Jennifer’s express app is pretty simple, the part to take a look a there is the ${api_key}. This is value is being set from an environment variable.

   const express = require('express');
   const fetch = require('node-fetch');
   const path = require('path');
   const app = express();
   const PORT = 80;
   app.use(express.static(path.join(__dirname, '.')));
   app.get('/giphy', async (req, res) => {
    try {
        const apiKey = process.env.GIPHY_API;
        const response = await fetch(`https://api.giphy.com/v1/gifs/random?api_key=${apiKey}&tag=cowboy`);
        const data = await response.json();
        res.json({ imageUrl: data.data.images.downsized_large.url });
    } catch (err) {
        res.status(500).send('Failed to fetch GIPHY.');
    }
    });
   
   app.listen(PORT, () => {
     console.log(`Server running on port ${PORT}`);
   });
   

   • The GIPHY_API environment variable is being configured by the Vault Agent sidecar that is configured as part of the app deployment. In this example it is provided as a single yaml but it could be injected as part of a deployment pipeline, or operator as well.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
    name: myapp
   spec:
   replicas: 1
   selector:
    matchLabels:
      service: myapp
      app: myapp
    template:
      metadata:
      labels:
        service: myapp
        app: myapp
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: myapp
        vault.hashicorp.com/agent-inject-secret-config: 'secrets/myapp/giphy'
        vault.hashicorp.com/agent-inject-template-config: |
          {{- with secret "secrets/myapp" -}}
            export GIPHY_API="{{ .data.giphy }}"
          {{- end }}        
   

2. Adi - The Platform Engineer

   • Adi streamlines the management of infrastructure and tools using Terraform, making it easy to add new features as needed.
   • Modular Terraform modules simplify platform setup and maintenance.

   module "gke_cl" {
    source             = "[email protected]:marc-leblanc/terraform-google-gke.git?ref=1.6.0"
    cluster_name       = "cl1-ce1"
    project_id         = var.gcp_project_id
    initial_node_count = "1"
    location           = var.gcp_region
    network            = google_compute_network.gcp_vpc.id
    machine_type       = "n2-standard-2"
   }
   
   module "consul" {
    source                   = "[email protected]:marc-leblanc/terraform-gke-consul.git?ref=1.0.2"
    consul_manage_acls       = true
    consul_prometheus_enable = true
    grafana_enable           = true
    prometheus_svc_type      = "LoadBalancer"
    grafana_svc_type         = "LoadBalancer" 
   }
   

3. Hannah - The SRE

   • With Prometheus and Grafana integrated through Consul, Hannah gains instant access to valuable application metrics.
   • The platform’s self-configuring nature reduces the need for manual intervention, enabling proactive monitoring.
   • Similar to the Developer, our SRE hero has a much easier and streamlined path to getting data. This is handled thanks to the use of Consul and annotations it automatically applies when an app joins the mesh.

   k describe pod myapp
   
   Namespace:          myapp
   Service Account:    myapp
   Labels:             app=myapp
   Annotations:        consul.hashicorp.com/connect-inject-status: injected
                       consul.hashicorp.com/connect-k8s-version: v1.2.2
                       consul.hashicorp.com/connect-service-port: 80
                       prometheus.io/path: /metrics
                       prometheus.io/port: 20200
                       prometheus.io/scrape: true
   status:             Running
   

4. Security Engineer - The Security Guru

   • Security policies are managed as code using Terraform, ensuring consistency and auditability.
   • Vault’s Kubernetes authentication provides a seamless way to secure applications without manual interference.
   • Defining a secret mount point in Vault using Terraform might look like this

   resource "vault_mount" "secret" {
    path      = "secrets"
    type      = "kv"
    options = {
      version = "1"
    }
   }
   

   • Creating a policy to read that mount point might look like this

   resource "vault_policy" "myapp-read" {
    name = "myapp-rt"
    policy = <<EOT
    path "secrets/myapp" {
     capabilities = ["read"]
    }
    EOT
   }
   

• Configuring the Vault Kubernetes auth engine to tie this all together in Terraform might look like this. This is allowing a pod running in namespace myapp as ServiceAccount myapp will be authenticated and associated with the myapp policy defined above

resource "vault_auth/kubernetes_role" "myapp" {
  name                     = "myapp"
  bound_service_account_names      = ["myapp"]
  bound_service_account_namespaces = ["myapp"]
  policies                  = ["myapp"]
  ttl                       = "1h"
}

Stack Summary

• Jennifer - The Developer:
  • Pain Point: Frequent context switching and becoming a security expert.
  • Solution: Focuses on writing code without needing to be a security expert. Vault provides seamless access to secure resources.
• Khalid - The Security Guru:
  • Pain Point: Overwhelming influx of security data and last-minute security adjustments.
  • Solution: Security policies managed as code with Terraform for consistency. Vault’s Kubernetes authentication ensures seamless application security.
• Hannah - The SRE:
  • Pain Point: Inconsistent data and disjointed tools.
  • Solution: Access to valuable metrics through Prometheus and Grafana via Consul. Self-configuring platform reduces manual intervention for proactive monitoring.
• Adi - The Platform Engineer:
  • Pain Point: Constant firefighting and disruptions, managing multiple tools.
  • Solution: Streamlines infrastructure and tool management with Terraform. Modular Terraform modules simplify platform setup and maintenance.

Conclusion

In conclusion, the journey from friction to flow in teamwork hinges on the ability to empathize with the diverse perspectives, priorities, and experiences that each member of a cross-functional team brings. It’s crucial to recognize that while we have a shared goal of delivering successful outcomes, becoming an expert in everyone’s field is not the objective.

By taking into consideration the unique perspectives and pain points of each persona within your team and implementing purposeful platform solutions, you can elevate teamwork and create a more efficient and satisfying development process. This approach fosters a harmonious and productive team environment where experts can focus on their strengths, eliminating unnecessary roadblocks, and ultimately delivering happiness to your customers.

Code References

You can browse some of the code used in this blog here. They may be good starting points for you to dabble in a lab.

Recording

Prefer video to all this reading? You can find the recorded HashiConf Global 23' talk on Youtube.

Banner image - Photo by Mike Lewis HeadSmart Media on Unsplash

• DevOps
• Culture
• Platform Engineering
• Happy People
• Terraform
• IAC
• Vault
• Consul