` to say “The Awesome Todo App”). 5. Build the Docker image now using `docker build -t getting-started .` again. This time, your output should look a little different. ``` Sending build context to Docker daemon 219.1kB Step 1/6 : FROM node:12-alpine ---> b0dc3a5e5e9e Step 2/6 : WORKDIR /app ---> Using cache ---> 9577ae713121 Step 3/6 : COPY package.json yarn.lock ./ ---> Using cache ---> bd5306f49fc8 Step 4/6 : RUN yarn install --production ---> Using cache ---> 4e68fbc2d704 Step 5/6 : COPY . . ---> cccde25a3d9a Step 6/6 : CMD ["node", "src/index.js"] ---> Running in 2be75662c150 Removing intermediate container 2be75662c150 ---> 458e5c6f080c Successfully built 458e5c6f080c Successfully tagged getting-started:latest ``` First off, you should notice that the build was MUCH faster! And, you’ll see that steps 1-4 all have `Using cache`. So, hooray! We’re using the build cache. Pushing and pulling this image and updates to it will be much faster as well. Hooray! ## Multi-stage builds While we’re not going to dive into it too much in this tutorial, multi-stage builds are an incredibly powerful tool to help use multiple stages to create an image. There are several advantages for them: * Separate build-time dependencies from runtime dependencies * Reduce overall image size by shipping *only* what your app needs to run ### Maven/Tomcat example When building Java-based applications, a JDK is needed to compile the source code to Java bytecode. However, that JDK isn’t needed in production. Also, you might be using tools like Maven or Gradle to help build the app. Those also aren’t needed in our final image. Multi-stage builds help. ``` # syntax=docker/dockerfile:1 FROM maven AS build WORKDIR /app COPY . . RUN mvn package FROM tomcat COPY --from=build /app/target/file.war /usr/local/tomcat/webapps ``` In this example, we use one stage (called `build`) to perform the actual Java build using Maven. In the second stage (starting at `FROM tomcat`), we copy in files from the `build` stage. The final image is only the last stage being created (which can be overridden using the `--target` flag). ### React example When building React applications, we need a Node environment to compile the JS code (typically JSX), SASS stylesheets, and more into static HTML, JS, and CSS. If we aren’t doing server-side rendering, we don’t even need a Node environment for our production build. Why not ship the static resources in a static nginx container? ``` # syntax=docker/dockerfile:1 FROM node:12 AS build WORKDIR /app COPY package* yarn.lock ./ RUN yarn install COPY public ./public COPY src ./src RUN yarn run build FROM nginx:alpine COPY --from=build /app/build /usr/share/nginx/html ``` Here, we are using a `node:12` image to perform the build (maximizing layer caching) and then copying the output into an nginx container. Cool, huh? # Docker Development ## How to keep your images small Small images are faster to pull over the network and faster to load into memory when starting containers or services. There are a few rules of thumb to keep image size small: * Start with an appropriate base image. For instance, if you need a JDK, consider basing your image on the official `openjdk` image, rather than starting with a generic `ubuntu` image and installing `openjdk` as part of the Dockerfile. * [Use multistage builds](https://docs.docker.com/build/building/multi-stage/). For instance, you can use the `maven` image to build your Java application, then reset to the `tomcat` image and copy the Java artifacts into the correct location to deploy your app, all in the same Dockerfile. This means that your final image doesn’t include all of the libraries and dependencies pulled in by the build, but only the artifacts and the environment needed to run them. * If you need to use a version of Docker that does not include multistage builds, try to reduce the number of layers in your image by minimizing the number of separate `RUN` commands in your Dockerfile. You can do this by consolidating multiple commands into a single `RUN` line and using your shell’s mechanisms to combine them together. Consider the following two fragments. The first creates two layers in the image, while the second only creates one. ``` RUN apt-get -y update RUN apt-get install -y python ``` ``` RUN apt-get -y update && apt-get install -y python ``` * If you have multiple images with a lot in common, consider creating your own [base image](https://docs.docker.com/develop/develop-images/baseimages/) with the shared components, and basing your unique images on that. Docker only needs to load the common layers once, and they are cached. This means that your derivative images use memory on the Docker host more efficiently and load more quickly. * To keep your production image lean but allow for debugging, consider using the production image as the base image for the debug image. Additional testing or debugging tooling can be added on top of the production image. * When building images, always tag them with useful tags which codify version information, intended destination (`prod` or `test`, for instance), stability, or other information that is useful when deploying the application in different environments. Do not rely on the automatically-created `latest` tag. ## Where and how to persist application data * **Avoid** storing application data in your container’s writable layer using [storage drivers](https://docs.docker.com/storage/storagedriver/select-storage-driver/). This increases the size of your container and is less efficient from an I/O perspective than using volumes or bind mounts. * Instead, store data using [volumes](https://docs.docker.com/storage/volumes/). * One case where it is appropriate to use [bind mounts](https://docs.docker.com/storage/bind-mounts/) is during development, when you may want to mount your source directory or a binary you just built into your container. For production, use a volume instead, mounting it into the same location as you mounted a bind mount during development. * For production, use [secrets](https://docs.docker.com/engine/swarm/secrets/) to store sensitive application data used by services, and use [configs](https://docs.docker.com/engine/swarm/configs/) for non-sensitive data such as configuration files. If you currently use standalone containers, consider migrating to use single-replica services, so that you can take advantage of these service-only features. # CLI Cheat Sheet <figure><img src="/files/n7jY6dPMGdDvXRJqmhNc" alt=""><figcaption></figcaption></figure> # Container Orchestration # Kubernetes Kubernetes is an open source, portable platform designed for managing containerized workloads and services through declarative configuration and automation. The term "Kubernetes" comes from the Greek word for helmsman or pilot, and the abbreviation "K8s" stems from counting the eight letters between "K" and "s." Originating as a Google project in 2014, Kubernetes integrates more than 15 years of Google's expertise in handling large-scale production workloads with community-driven best practices. <figure><img src="/files/IHwywrPT7Tqd4fU4UI09" alt="" width="563"><figcaption></figcaption></figure> ## History ### **Traditional deployment era** Early on, organizations ran applications on physical servers and there was no way to define resource boundaries for applications. This caused issues where one application would take up most of the resources. ### **Virtualized deployment era** As a solution, virtualization was introduced. It allowed better utilization of resources in a physical server and security. ### **Container deployment era** Containers are comparable to virtual machines but with relaxed isolation properties, shared operating system, making them lightweight and portable across clouds and OS distributions, gaining popularity for agile application creation, continuous development, DevOps separation, observability, environmental consistency, cloud and OS distribution portability, application-centric management, and resource efficiency. <figure><img src="/files/HDigXge77XFp97L4Ygc5" alt="" width="563"><figcaption>History Visualized</figcaption></figure> \ # Benefits **Service Discovery and Load Balancing:** * Expose containers using DNS names or IP addresses. * Built-in load balancing for stable deployment. **Storage Orchestration:** * Automatically mount various storage systems (local storage, public cloud providers). **Automated Rollouts and Rollbacks:** * Define and automate the desired state of deployed containers. * Facilitate controlled updates and rollbacks. **Automatic Bin Packing:** * Efficiently allocate CPU and memory resources. * Optimize container placement on a cluster of nodes. **Self-Healing:** * Kubernetes restarts, replaces, and manages containers in response to failures. * Ensure continuous service availability. **Secret and Configuration Management:** * Safely store and update sensitive information. * Without rebuilding container images or exposing secrets. **Batch Execution:** * Handle batch and CI workloads. * Replace failed containers as needed. **Horizontal Scaling:** * Easily scale applications up or down based on demand. * Manual, UI-based, or automated CPU usage-triggered scaling. **IPv4/IPv6 Dual-Stack:** * Allocate both IPv4 and IPv6 addresses to Pods and Services. * Comprehensive networking support. **Designed for Extensibility:** * Extend Kubernetes clusters with additional features. * Without modifying upstream source code. # Cheat Sheet {% file src="/files/qq3hr6lyMO4PgeTq64V4" %} # Components A Kubernetes **cluster** is a **set of nodes** that run containerized applications. It comes with something called a **control plane** that manages the overall state of the cluster, as well as make global decisions about the cluster (for example, scheduling). It is responsible for maintaining the desired state of the cluster, responding to user requests, and managing the deployment and scaling of applications. ## Control Plane Components <figure><img src="/files/IsTODGQw0WSHvYaOcgZm" alt=""><figcaption><a href="https://kubesphere.io/blogs/monitoring-k8s-control-plane/">Reference</a></figcaption></figure> ### **API Server** * The API server is a component that exposes the Kubernetes API. It serves as the front end for the control plane. Users, other components, and external tools communicate with the cluster through the API server. ### **etcd** * etcd is an open source distributed key-value store that stores the configuration data of the cluster, representing the overall state of the system. The control plane components watch for changes in etcd and react accordingly to maintain the desired state. ### **Controller Manager** * The controller manager is responsible for running controller processes that regulate the state of the cluster. Examples include the Replication Controller, which ensures the correct number of replicas for a set of pods, and the Node Controller, which monitors and responds to changes in the nodes of the cluster. ### **Scheduler** * The scheduler is responsible for placing pods onto nodes in the cluster. It takes into account factors such as resource requirements, hardware constraints, and affinity/anti-affinity specifications when making placement decisions. These components work together to ensure that the cluster is in the desired state, and they continuously monitor and adjust the cluster to maintain that state. The control plane components are typically distributed across multiple nodes for high availability and fault tolerance. ## Worker Node Components <figure><img src="/files/rjyckqMEXwjFjUUiDGBr" alt=""><figcaption><a href="https://itnext.io/one-story-for-effortlessly-passing-kubernetes-interview-questions-in-2023-f93e828cee9f">Reference</a></figcaption></figure> The worker nodes are responsible for hosting the application pods in the cluster. ### **Kubelet** * Kubelet is an agent that runs on each node and is responsible for creating the pods by the provided YAML specs, reporting the health status of the pods and providing status information on the node. (network, disk space, etc.) ### **Container Runtime** * The container runtime is the software responsible for running containers. Kubernetes supports various container runtimes, including Docker, containerd, and others. The container runtime is responsible for pulling container images from a container registry, creating containers, and managing their lifecycle. ### **Kube-proxy** * Kube-proxy is responsible for network proxying on the worker nodes. It maintains network rules to allow communication to Pods from network sessions inside or outside of the cluster. Kube-proxy enables the communication between different Pods and services in the cluster. # Pods **Pods** are the smallest deployable units of computing that you can create and manage in Kubernetes. A Pod is similar to a set of containers with shared namespaces and shared filesystem volumes. <figure><img src="/files/UgzSjIAKw546Mb8dAaxF" alt=""><figcaption></figcaption></figure> **Init containers** are executed before both the sidecar and main application containers, and their successful completion is a prerequisite for the other containers to start. Init containers offer versatility, allowing actions like checking for application dependencies or configuring the environment for main and sidecar containers. **Sidecar containers** run concurrently with the main application container and serve various purposes. For instance, in Istio, a sidecar container functions as a traffic proxy for the main container, and it can also handle tasks such as logging and monitoring. # Workload Resources In Kubernetes, applications run in containers within Pods. To simplify management, higher-level workload objects are created using the Kubernetes API. Key workload types include the Deployment, the StatefulSet and the DaemonSet. <figure><img src="/files/BzzYuNLcyn0Oclr98J15" alt=""><figcaption></figcaption></figure> **Deployment** is the go-to resource for easily deploying stateless applications in a Kubernetes cluster. It uses a ReplicaSet for straightforward version rollbacks. The pod naming convention is `<deployment-name>`-`<replicaset-id>`-`<pod-id>`. **StatefulSet**, introduced in Kubernetes 1.9, is designed for stable hosting of stateful applications. It independently manages pods and requires a headless service for network identification and DNS resolution. Pods in a StatefulSet have unique names like `<Statefulset-name>`-0, `<Statefulset-name>`-1, and each has its own persistence volume claim. StatefulSets are ideal for stateful applications, such as databases, where replica identification and graceful upgrades are essential. # Best Practices <https://aws.github.io/aws-eks-best-practices/> # Developer Portal 👨‍💻 ## Introduction In early 2023, our engineering teams were spending one to three weeks per month manually setting up new projects leading to significant productivity loss and inconsistent configurations. After doing a developer survey and quantifying the impact, I built a self-service developer portal using Backstage that reduced project setup time from one week to less than an hour. This presentation walks through the journey of designing and implementing this solution. ## Problem Space 🔍 * ⏰ **Time Cost**: \~1 week per project setup * 📈 **Monthly Impact**: 1-3 projects = Up to 3 weeks engineering time lost * 👥 **Scale**: Supporting 200+ engineers * 💸 **Business Cost**: \~$15,000 per project setup (assuming average engineer salary) #### Time Breakdown ⏱️ ``` Project Setup Timeline: ├── Infrastructure Setup: 2-3 days ├── Access Management: 1 day ├── Platform Integration: 1-2 days └── Testing & Validation: 1 day ``` ## Pain Points Identified Through Survey <figure><img src="/files/YvJMmUbqq8IIwogGvHEy" alt=""><figcaption>Manual Project Setup Flow</figcaption></figure> * **Infrastructure Challenges** 🏗️ * Engineers lacking IaC knowledge and taking time to understand guidelines * Limited terraform expertise in product teams * **Repository Setup Issues** 🔐 * Manual repository creation * Time-consuming GitHub team setup * Inconsistent repository settings * **Platform Integration Friction** 🔄 * Context switching between platforms: * 🔍 Sentry for error tracking * 📊 Datadog for monitoring * 🚦 LaunchDarkly for feature flags * [️🚨](https://emojis.wiki/security/) Snyk for security scanning * Manual resource creation in each platform # Solution Overview 🎯 ## Vision Build a self-service developer portal that automates project creation and standardizes infrastructure setup across the organization. ## Core Requirements ### **Functional Requirements** * **Project Creation** * Create new projects from templates * Configure project CI/CD and infrastructure * Automate repository setup and permissions * Set up required platform integrations (Sentry, Datadog, etc.) * **Platform Features** * Centralized API documentation * Service discovery ### **Non-functional Requirements** * **Performance** * Reduce project setup time from 1 week to under 2 hours * Handle concurrent project creation requests * **Scalability** * Support 200+ engineers * Handle multiple project templates * Scale across different engineering teams * **Security** * Enterprise SSO Integration * Role based access control * **Maintainability** * Maintainable template system * Standardized infrastructure patterns * Clear documentation * **Reliability** * Consistent project creation * Error handling and recovery ## Why Backstage? * Wrote a tech brief that included a decision matrix with the popular Developer Portal platforms * Backstage emerged as the optimal choice based on: * Open source and free * Great documentation * Intuitive UI/UX * 100+ existing plugins * Strong developer experience focus {% embed url="<https://youtu.be/85TQEpNCaU0>" %} # System Architecture 🏗️ Intro ## Infrastructure Overview In this section, I will delve into an example implementation of Backstage using a combination of Google Cloud Platform (GCP) services, Okta for Single Sign-On (SSO) authentication and authorization, and GitHub integration to access organization data. Here's a preliminary glimpse into the system's structure: <figure><img src="/files/QvwE0llebILCkO5S4DBq" alt=""><figcaption></figcaption></figure> * **Backstage Layer** * Backstage deployed on Cloud Run * Traffic outside of work hours is very low to none * ✅ Cost savings from going serverless and scaling to 0 instances when not in use * ⚠️ Tradeoff was cold starts. The first request needs to wait for a new container, causing a delay (10-15s). * Cloud Run has [autoscaling](https://cloud.google.com/run/docs/about-instance-autoscaling), meaning that you don't necessarily need to put a load balancer in front! * **Storage Layer** * Cloud SQL: User, project and organization metadata * Cloud Storage: API documentation stored (swagger/openAPI) * Secret Manager: App secrets fetched and used as environment variables during build and runtime * **Integration Layer** * Okta: Authentication/Authorization * GitHub: Repository and organization data management * Third-party tools: Monitoring via Cloud Logs and Google Analytics. * **Nice-to-Have (But Not Achieved)** * Leveraging [cache](https://backstage.io/docs/overview/architecture-overview/#cache) stores (Redis, etc.) to improve performance ### CI/CD Deployment Pipeline 🔄 <figure><img src="/files/Xel4xINcJQSZWWfZmTrd" alt=""><figcaption></figcaption></figure> **Pipeline Components** * **Trigger**: Cloud Build trigger listening for merge to default branch events * **Build Process**: * Pull latest Backstage image from Artifact Registry (build cache) * Build new Docker image with latest changes * Push updated image to Artifact Registry * Deploy new instance to Cloud Run * **Nice-to-Have (But Not Achieved):** * Staging environment * Canary rollout * Small percentage -> monitor metrics -> increase traffic if stable **Key Benefits** * **Efficient Builds**: Docker layer caching reduces build time * **GitOps Deployment**: Zero-touch deployment process * **Rollback Capability**: Easy rollback to previous versions # Implementation Journey 🛠️ ## Phase 0: POC (Weeks 1 & 2) * **Goals** * Local Backstage setup * Quick example template to showcase automation capabilities * Service Discovery example * read Org data from Github * Scaffolder Automation example * write to Github (pull requests & repository creation) * **Challenges** * Learning about platform and configuration * Getting permissions to create a Github App * **Outcomes** * Transferred knowledge to team * Demo'd platform capabilities * Received approval to implement ## Phase 1: Deployment, Service Discovery & API Docs (Quarter 1) * **Goals** * Deploy Backstage to Cloud * Authentication via Okta SSO integration * API Documentation Feature Setup * **Challenges** * IaC and CI/CD setup * Multiple meetings with Cloud Engineering team to figure out right architecture * Settled on Cloud Run for cost savings * GitOps CI/CD Setup * Deployments on merge to default branch * Automated tests (linting, unit, e2e, etc.) on push to any branch * Authentication requirements gathering and implementation * Worked with IT to receive secrets and setup Okta configuration * API documentation publishing feature released * Friction to push teams to use feature * Cross-team alignment on service discovery standards * Once again friction to get teams to implement across org repositories * **Outcomes** * First few Backstage deployments * Secure authentication flow * CI/CD pipeline setup * Centralized API documentation started * Found out it takes time to get adoption :crying\_cat\_face: * Initial developer feedback ## Phase 2: Scaffolder Templates (Quarter 2) * **Goals** * Achieve full API documentation adoption * Scaffolder templates framework * Templates for backend and frontend engineers * Plugins to reduce context switching * Google Analytics to track usage of Backstage * **Challenges** * Gathering unwritten engineering standards * Working with architecture team and engineers to document best practices and create re-usable "skeleton" repositories * Long hours spent testing and debugging templates * Implementing plugins for various patforms (datadog, sentry, snyk and entity feedback) * **Outcomes** * Crentralized all of our API docs * Added two backend templates (nodejs/php microservice) and one frontend template (vuejs microfrontend) * Documented engineering standards for all future projects * Quick wins implementing plugins that reduced context switching and allowed engineers to leave feedback on eachother's projects ## Phase 3: More Templates & Enhancements (Quarter 3-4) * **Goals** * Work with new teams to create more templates * Implement cost insights * Automate engineering standard checks against entities and visualization * **Challenges** * Org wide tagging of cloud resources to allow getting GCP billing data linked to Backstage entities * Took a lot of effort because it affected lots of team's projects IaC * Migrating engineering standards documentation to automated checks using Spotify's Sound Check plugin * Lots of time spent figuring out how to get the correct metadata to make sure checks are reliable * **Outcomes** * Multiple team-specific and generic templates like pub/sub topics * Users can visualize cloud costs per entity <figure><img src="/files/wdCZiz1rXwKHM08xpLHu" alt=""><figcaption></figcaption></figure> * Engineering standards turned in to automated checks which allowed visualizing service maturity * Management can track reliability and quality insights on their team's projects <figure><img src="/files/sUnmN6eqlYiGYa5Q9oKo" alt=""><figcaption></figcaption></figure> # Cross-team Collaboration 🤝 ## Stakeholder Engagement * **Product Engineering Teams** * API documentation requirements * Template feedback * Knowledge transfer and pair programming to deliver new templates * **Cloud Engineering** * Infrastructure templates (Terraform modules) * Security requirements (IAM, Service Accounts) * Resource provisioning (Atlantis/Terragrunt) * **Architecture Team** * Design patterns * Example: Modular templating pattern * Engineering standards * Example: Naming conventions, GitOps deployment pattern, Code Quality, Observability, Monitoring, etc. * **IT Department** * Authentication setup * Access management * Compliance requirements ## Communication Channels * Weekly sync meetings * Template review process * Feedback collection system # Lessons & Future 🎓 ## Key Learnings * Start with high-impact features * Friction is necessary when setting standards * Regular stakeholder alignment is needed ## Future Roadmap * Usage Insights <div align="left"><figure><img src="/files/xAdcgOmQDPMD60DhIDWo" alt="" width="375"><figcaption></figcaption></figure></div> * RBAC to control access to data and actions ## Challenges Overcome * Cross-team coordination and alignment * Competing priorities are hard to navigate * Communicating big changes to processes * Documentation, workshops, demos and more * Permission complexity * IAM and service accounts are tricky sometimes or maybe just in GCP * Template standardization * Different teams have unique needs so good customization options were necessary # Provisioning # Terraform Infrastructure as code (IaC) tools allow you to manage infrastructure with configuration files rather than through a graphical user interface. IaC allows you to build, change, and manage your infrastructure in a safe, consistent, and repeatable way by defining resource configurations that you can version, reuse, and share. Terraform is **HashiCorp's** infrastructure as code tool. It lets you define resources and infrastructure in human-readable, declarative configuration files, and manages your infrastructure's lifecycle. Using Terraform has several advantages over manually managing your infrastructure: * Terraform can manage infrastructure on multiple cloud platforms. * The human-readable configuration language helps you write infrastructure code quickly. * Terraform's state allows you to track resource changes throughout your deployments. * You can commit your configurations to version control to safely collaborate on infrastructure. Terraform's configuration language is declarative, meaning that it describes the desired end-state for your infrastructure, in contrast to procedural programming languages that require step-by-step instructions to perform tasks. Terraform providers automatically calculate dependencies between resources to create or destroy them in the correct order. # Installation [Homebrew](https://brew.sh/) is a free and open-source package management system for Mac OS X. Install the official [Terraform formula](https://github.com/hashicorp/homebrew-tap) from the terminal. First, install the HashiCorp tap, a repository of all our Homebrew packages ``` brew tap hashicorp/tap ``` Now, install Terraform with `hashicorp/tap/terraform`. ``` brew install hashicorp/tap/terraform ``` Install autocompletion ``` terraform -install-autocomplete ``` Update Terraform ``` brew upgrade terraform ``` # Usage ## [init](https://www.terraform.io/cli/commands/init) ``` terraform init ``` It’s  the  first  command  you  need  to  execute.  Otherwise,  terraform plan,  apply,  destroy  and  import  will  not  work. It is safe to run this command multiple times.  The  command will  : * find terraform  modules based on source provided * configure and validate  a  backend (if present) * install provider(s)  plugins ## [validate](https://www.terraform.io/cli/commands/validate) ``` terraform validate ``` Once you’ve initialized the directory, it’s a good idea to run the`validate` command before you run `plan` or `apply`. Validation catches syntax errors, version errors and more. > It is safe to run this command automatically, for example as a post-save check in a text editor or as a test step for a re-usable module in a CI system. ## [get](https://www.terraform.io/cli/commands/get) This  command  is  useful  when you have  some  modules defined. ```  terraform get  ``` The modules are downloaded into a `.terraform` subdirectory of the current working directory. **Don't commit this directory to your version control repository.** > `update=true` - If specified, modules that are already downloaded will be checked for updates and the updates will be downloaded if present. ## [plan](https://www.terraform.io/cli/commands/plan) The  plan  command creates an execution plan which allows previewing changes. Like a dry run, it won't actually apply any of the proposed changes. ```  terraform  plan  ``` > By default, when Terraform creates a plan it: > > * Reads the current state of any already-existing remote objects to make sure that the Terraform state is up-to-date. > * Compares the current configuration to the prior state and noting any differences. > * Proposes a set of change actions that should, if applied, make the remote objects match the configuration. ## [apply](https://www.terraform.io/cli/commands/apply) Next comes the apply command which will execute the actions proposed in the plan. ``` terraform apply ``` > `-auto-approve` - Skips interactive approval of plan before applying. This option is ignored when you pass a previously-saved plan file, because Terraform considers you passing the plan file as the approval and so will never prompt in that case. Apply only one resource ``` terraform  apply -target="module.s3" ``` ## [destroy](https://www.terraform.io/cli/commands/destroy) ```  terraform  destroy ``` This command deletes all the resources in the configuration. A  deletion  plan  can  be  created  before execution: ```  terraform  plan  –destroy ``` Deletions can take a target resource too. ``` terraform  destroy - target=" aws_s3_bucket.my_bucket" ``` # Configuration Management In DevOps, **configuration management** is a practice that involves **tracking** and **managing configurations** of infrastructure and software to ensure **consistency** and **reliability** across different environments. It relies on **automation**tools to streamline setup and updates, and **version control** systems to manage changes and enable rollbacks. Additionally, it supports **compliance** and **auditing** by monitoring configurations against organizational policies and providing an audit trail. # Ansible Ansible is an open-source configuration management, application deployment and provisioning tool that uses its own declarative language in YAML. Ansible is agentless, meaning you only need remote connections in order for it to function. [**Official Documentation**](https://docs.ansible.com/) # Benefits Ansible, a cutting-edge automation tool, proves invaluable for companies seeking efficient task and command automation across diverse nodes, including servers and PCs. Its versatility extends to both on-premises and cloud environments. ## Understanding Ansible in the Ecosystem * **Configuration Management vs. Provisioning:** While Ansible excels in configuration management, Terraform serves as a provisioning tool, creating and managing Cloud Infrastructure resources. * **TLDR:** Provision with Terraform. Configure with Ansible. ## Key Features * **Open Source and Community Support:** Ansible is freely available as an open-source tool (GitHub repository: [ansible/ansible](https://github.com/ansible/ansible)), boasting a robust and expansive community. * **Infrastructure as Code (IaC):** Commands, tasks, and codes seamlessly transition into Infrastructure as Code, enabling tasks to be saved, versioned, repeated, and tested. * **Declarative Configuration:** IaC adopts a 'Declarative Way,' allowing the definition of the desired configuration for streamlined management. * **Agentless Architecture:** Unlike traditional approaches, Ansible operates without requiring an agent app on the worker node. * **Parallel Execution:** Ansible inherently performs tasks in parallel across multiple hosts, optimizing efficiency. * **Cross-Platform Compatibility:** Supporting both Linux and Windows PCs, Ansible proves versatile in addressing various operating systems. * **Comprehensive Documentation:** Ansible offers well-designed documentation for users, facilitating a smoother learning and implementation process ([Ansible Documentation](https://docs.ansible.com/)). * **SSH Communication:** Ansible leverages SSH for seamless communication with other nodes. * **Module-Based Task Handling:** The tool efficiently manages tasks through a modular approach, enhancing its capability to handle diverse tasks. ## Integration with Other Technologies * **Integration with Terraform:** Ansible seamlessly integrates with various technologies, such as Terraform, expanding its capabilities and adaptability. In summary, Ansible stands out for its user-friendly and powerful automation capabilities, with seamless integration possibilities, making it a preferred choice for diverse tasks in both on-premises and cloud environments. # Installation {% embed url="<https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-and-upgrading-ansible-with-pipx>" %} Pipx installs and runs Python applications in isolated environments. ```bash brew install pipx pipx ensurepath pipx install --include-deps ansible # full version pipx install ansible-core # minimal version ``` At this point you should be able to get the version of ansible in a new shell session. ```bash ansible --version # check if it worked ``` # Build Systems # Bazel Intro Bazel is an **opinionated** **build system** that I’ve come to appreciate for its efficiency and scalability. Originally **developed by Google** to tackle the challenge of managing extensive codebases across various languages and platforms, Bazel is now available as **open-source**. Official documentation can be found here -> <https://docs.bazel.build> In the subsections, I will cover concepts I've learned and provide tips. # Features <details> <summary>Explicit Dependency Management</summary> #### What is Explicit Dependency Management? In Bazel, **Explicit Dependency Management** means that you must clearly and explicitly list all the dependencies that your build targets need. Unlike some other build systems where dependencies might be inferred automatically, Bazel requires you to specify each dependency directly. The dependency graph in Bazel must always be a **directed acyclic graph** which means no cycles are allowed. #### Basic Idea * **Clear Declaration:** You must list every file, library, or target your build depends on in your build configuration files. This ensures that Bazel knows exactly what each target needs to build correctly. * **No Implicit Dependencies:** There is no automatic assumption about what dependencies a target might need based on file locations or other heuristics. Everything must be explicitly stated. #### Example Scenario Let’s say you’re working on a project with a library and an application that depends on that library. Here’s how you’d manage dependencies in Bazel: 1. **Project Structure:** ```less lessCopy codemy_project/ ├── //app │ ├── BUILD.bazel │ └── main.py └── //lib ├── BUILD.bazel └── utils.py ``` 2. **Targets and Dependencies:** * **`//lib:utils`** - A target that builds a library with utility functions. * **`//app:main`** - A target that builds an application and depends on the utility functions. 3. **Setting Up Explicit Dependencies:** **`//lib/BUILD.bazel`** ```python py_library( name = "utils", srcs = ["utils.py"], ) ``` **`//app/BUILD.bazel`** ```python py_binary( name = "main", srcs = ["main.py"], deps = ["//lib:utils"], # Explicitly declare the dependency on //lib:utils ) ``` #### Explanation * **Explicit Declaration:** * In the `//lib/BUILD.bazel` file, the `py_library` target `//lib:utils` is defined with its source files. This target represents the library with utility functions. * In the `//app/BUILD.bazel` file, the `py_binary` target `//app:main` is defined as a Python binary. The `deps`attribute explicitly lists `//lib:utils` as a dependency. This means that `//app:main` requires `//lib:utils` to build. * **No Implicit Dependencies:** * Bazel doesn’t guess or infer that `//app:main` might need `//lib:utils` based on file paths or other heuristics. You must explicitly state this dependency in the `deps` attribute. #### Benefits * **Predictability:** By explicitly declaring all dependencies, you ensure that your build is predictable and reproducible. You know exactly what each target depends on. * **Clarity:** It makes the build configuration clear and understandable. Anyone looking at the build files can see all the dependencies listed without guessing. * **Error Prevention:** It reduces the risk of build errors due to missing dependencies. Since everything is declared explicitly, Bazel will catch any missing dependencies and prompt you to fix them. #### Comparison to Implicit Dependencies * **Implicit Dependencies:** In some build systems, dependencies might be inferred based on file locations or naming conventions. This can lead to unexpected issues if the system makes incorrect assumptions. * **Explicit Dependencies (Bazel):** Bazel’s requirement for explicit declarations avoids such issues. You specify exactly what each target needs, which makes the build process more reliable. #### Summary **Explicit Dependency Management** in Bazel means you must clearly list every dependency your build targets require. This approach ensures that your build process is predictable, clear, and free from implicit assumptions about what dependencies are needed. By specifying dependencies directly in your build configuration files, you avoid potential issues and make your project easier to manage. </details> <details> <summary>Advanced Visibility Features</summary> #### What is Visibility in Bazel? In Bazel, visibility is a feature that allows you to control which parts of your code can access or depend on specific build targets. This helps you manage and restrict how different parts of your project interact with each other, improving modularity and reducing the risk of unintended dependencies. #### Basic Idea * **Visibility Control:** You can limit which packages or targets are allowed to depend on a particular target. This way, you can keep certain parts of your codebase hidden from others and only expose what's necessary. #### Example Scenario Imagine you have a project with multiple modules: 1. **Project Structure:** ```less my_project/ ├── //frontend │ ├── BUILD.bazel │ └── main.py ├── //backend │ ├── BUILD.bazel │ └── service.py └── //shared ├── BUILD.bazel └── utils.py ``` 2. **Targets and Visibility:** * **`//frontend:main`** - A target that builds the frontend module. * **`//backend:service`** - A target that builds the backend module. * **`//shared:utils`** - A target that builds some shared utility functions. 3. **Setting Up Visibility:** * You want to allow only the `frontend` module to use the utilities from `shared`, but you don’t want the `backend` module to have access to these utilities. Here’s how you can set it up in Bazel: **`//shared/BUILD.bazel`** ```python pythonCopy codepy_library( name = "utils", srcs = ["utils.py"], visibility = ["//frontend:__pkg__"], # Only the frontend module can use this target ) ``` **`//frontend/BUILD.bazel`** ```python pythonCopy codepy_binary( name = "main", srcs = ["main.py"], deps = ["//shared:utils"], # Can depend on utils ) ``` **`//backend/BUILD.bazel`** ```python pythonCopy codepy_binary( name = "service", srcs = ["service.py"], # No dependency on //shared:utils, as it's not listed here ) ``` #### Explanation * **Visibility Declaration:** * In the `//shared/BUILD.bazel` file, the `visibility` attribute for `//shared:utils` is set to `["//frontend:__pkg__"]`. This means that only targets in the `frontend` package can see and depend on the `//shared:utils` target. * **Controlled Access:** * The `frontend` module can use the `utils` library because it is explicitly allowed by the visibility setting. * The `backend` module cannot access the `utils` library because it’s not listed in the visibility setting. This prevents unintended dependencies and keeps the `backend` module isolated from the `shared` utilities. #### Benefits * **Modularity:** By controlling visibility, you can better organize your project and enforce modular design. Each part of the project only has access to the dependencies it needs. * **Maintainability:** It reduces the risk of changes in one part of the project affecting others unintentionally, making the project easier to maintain and refactor. #### Summary Bazel’s visibility feature lets you control which parts of your project can access specific build targets. By using visibility, you can limit dependencies and keep different parts of your codebase modular and isolated, similar to how you might use package visibility in Java or namespaces in C++. </details> **TODO:** * Remote Build Execution and Caching * Build Dependency Analysis * Fast, Correct Builds (and Tests) # Secure Software Engineering As a software engineer, I know how important it is to build secure software in today's world. The CSSLP training I've completed has given me the skills to create software that's safe from cyber threats.\ \ The content presented in the following sections is based on my learning from the "All-in-One is All You Need" CSSLP exam guide. \ \ This comprehensive guide has been an invaluable resource in my journey to understand and master the concepts of secure software development and secure software lifecycle management. It provides comprehensive insights into the CSSLP domains and serves as a foundation for my knowledge in this critical field. # Core Concepts The Secure Software Concepts domain encompasses essential principles from information security that are crucial for secure software development, including **confidentiality**, **integrity and** **availability** (CIA). To fully describe operational security elements, authentication, authorization, auditability and nonrepudiation have been incorporated into the existing CIA framework. * **Confidentiality**: Ensuring that information is only accessible to those with the proper authorization. * **Integrity**: Maintaining the accuracy and reliability of data and systems, preventing unauthorized tampering or alterations. * **Availability**: Ensuring that systems and data are consistently accessible and operational when needed. * **Authentication**: Verifying the identity of users or entities accessing a system or data. * **Authorization**: Granting or denying specific permissions and access rights to users or entities based on their authenticated identity. * **Auditability**: The capability to track and record actions and events within a system for monitoring and accountability purposes. * **Nonrepudiation**: Preventing individuals from denying their actions or transactions, ensuring accountability and trustworthiness. ### Secure Development Lifecycle Adding team awareness and education, gates and security requirements, bug tracking, threat modeling, fuzzing, security reviews and mitigations to the software development lifecycle makes it more secure. #### Team Awareness & Education All team members should undergo ongoing role-specific training in both basic security knowledge, essential for the entire team, and advanced topics tailored to their specific roles to ensure they remain up-to-date and equipped to address evolving security challenges throughout their careers. #### Gates & Security Requirements During the development process, periodic security reviews, known as gates, are conducted to ensure adherence to security requirements. #### Bug Tracking As code is being developed, it's common to uncover bugs, which are code components exhibiting problematic behaviors, occasionally presenting exploitable vulnerabilities, and necessitating tracking and resolution. #### Threat Modeling Threat modeling serves to comprehensively identify and describe system threats while recording mitigation strategies. #### Fuzzing Fuzzing is an automated testing technique involving the application of diverse inputs to an interface, examining the outputs for unexpected behaviors, commonly used by both security testers and hackers to discover vulnerabilities and potential exploits. #### Security Reviews Security reviews serve as checkpoints within the process to ensure these steps are not bypassed and are functioning correctly. #### Mitigations Adopting a risk assessment model like DREAD, which calculates risk as a product of impact and probability, aids in prioritizing bug fixes, with the mitigation process offering four standard options for addressing security bugs: **do nothing**, **warn the user**, **remove the problem**, and **fix the problem**. #### Software Security vs Quality High-quality software can be defect-free but may not necessarily be secure. Conversely, the critical concern is that if software lacks quality and contains defects, it is unlikely to be secure. <figure><img src="/files/Qkj4qMbcHV27caMzuzyL" alt=""><figcaption>Security vs Quality</figcaption></figure> # Security Design Principles Secure designs are built upon a solid groundwork of secure design principles: * **Good enough security** * There is an appropriate level of required security for systems * **Least privilege** * Users and applications should have only the necessary rights and privileges to perform their current tasks. * **Separation of duties** * Software components should require multiple conditions to pass before a task can be considered as complete. This ensures no single individual can abuse the system * **Defense in depth** * Defense in depth is a security principle involving the use of multiple overlapping layers of protection to enhance security, with the understanding that no single defense mechanism is foolproof, and the goal is to make compromising a system more costly and time-consuming for adversaries. * **Fail-safe** * Fail-safe, in the context of system design, means that when a system encounters a failure, it should transition to a secure and stable state. * **Economy of mechanism** * Systems and security mechanisms should be kept as simple as possible because complexity increases the potential for vulnerabilities and makes troubleshooting more challenging. * **Complete mediation** * Complete mediation is the security principle that requires continuous verification of a subject's authorization for each access request to an object and action. * **Open design** * Open design is the principle that a system's security should not rely on the secrecy of its design or algorithms but instead on elements like keys, making the security independent of the design's obscurity. * **Least common mechanism** * Least common mechanism is a design approach aimed at preventing unintentional information sharing among processes by minimizing shared components, favoring separate processes for distinct functions to enhance security. * **Psychological acceptability** * A security system should not burden users as it will cause them to work around security aspects. * **Weakest link** * Common point of failure of a system. A system is only as strong as the weakest link. * **Leverage existing components** * New components increase the chances of new vulnerabilities being added. Therefore, reusing components should decrease the chances. * **Single point of failure** * Software systems should not have a single point of failure. # Software Security Requirements Requirements are like the blueprint of software, setting expectations for how it should function and operate. ### Functional Requirements Functional Requirements in software development specify what the software should do or the functions it should perform. Here are a few standard functional requirements the DevOps team would have: * Deployment platform requirements * Database requirements * Disaster recovery/business continuity planning (DR/BCP) requirements * Infrastructure requirements **Example:** "The system must allow users to log in using a username and password, and upon successful login, they should have access to their personal account information." ### Non-Functional Requirements Non-Functional Requirements, on the other hand, define how the system should perform its functions. They describe the quality attributes of the system, such as performance, security, scalability, and usability. In the context of security, a non-functional requirement could be: **Example:** "The system must encrypt all sensitive user data, such as passwords and financial information, using industry-standard encryption algorithms (e.g., AES-256) to ensure data confidentiality." ### Operational and Deployment Requirements Operational and deployment requirements in software development pertain to the considerations of how software will operate and be deployed within an enterprise environment, where it often coexists with established technology standards, such as specific platforms, operating systems, and infrastructure components. ### Use Cases Use cases are a valuable method for expressing functional requirements in a way that developers and testers can easily understand. They represent specific instances of how the system should behave and are particularly useful for describing complex or unclear interactions with the system, ensuring that both software design and testing adequately address potential issues arising from poorly defined requirements. Example: In an e-commerce application, a use case might be "User places an order," which describes the step-by-step process of how a user selects products, adds them to the cart, provides shipping information, and completes the purchase. This use case helps clarify the functional requirements for this specific user interaction. # Compliance Standards and Policies Regulations and compliance refer to the rules and standards that govern various aspects of an enterprise's operations, including those related to software development. Compliance with these rules is essential because failure to adhere to them can result in financial penalties, increased scrutiny, future regulatory constraints, and negative publicity. Compliance is distinct from security, as it focuses on adhering to established rules, even though strict compliance does not necessarily guarantee security. ## Security standards organizations ### ISO ISO stands for the International Organization for Standardization. It is an international organization that develops and publishes standards to ensure the quality, safety, efficiency, and interoperability of products, services, and systems across various industries, including information security. ### IEC The International Electrotechnical Commission (IEC) is an international standards organization that specializes in developing and publishing standards related to electrical and electronic technologies, components, and systems. #### ISO/IEC 9126 Quality Characteristics This ISO/IEC standard defines six quality characteristics that can be used to measure the quality of software: * Functionality * Reliability * Usability * Efficiency * Maintainability * Portability ### SAFECode SAFECode is an industry-supported organization that promotes collaboration among companies to enhance software assurance practices by sharing successful best practices and providing guidance on secure software development. [**Practical Security Stories**](http://safecode.org/wp-content/uploads/2018/01/SAFECode_Agile_Dev_Security0712.pdf) ### **OWASP** OWASP is a global, open community dedicated to enhancing application software security by publishing Top Ten vulnerability lists and providing valuable resources on their website ([www.owasp.org](http://www.owasp.org/)) to help organizations create more secure software. ### NIST NIST, or the National Institute of Standards and Technology, is a federal agency responsible for developing technology, measurements, and standards **aligned with the U.S. economy's interests**. \ \ Within NIST, the Computer Security Division addresses computer security concerns, including compliance with laws like the Federal Information Security Management Act (FISMA). NIST publishes key document types like Federal Information Processing Standards (FIPS) and the Special Publication (SP) 800 series, providing guidelines for information security practices, cryptographic protocols, and risk management frameworks, making them valuable resources for the information systems community Some important publications are: * FIPS 200 * FIPS 199 * FIPS 197 * FIPS 186-3 * FIPS 190-4 * FIPS 140 Series * SP 800-152 * SP 800-107 * SP 800-100 * SP 800-63 * SP 800-53 * SP 800-30 * SP 800-12 * SP 800-218 (SSDF) # Sarbanes-Oxley (SOX) In summary, a security software engineer should know that SOX, particularly Section 404, places significant emphasis on the security of financial reporting systems. They should understand the requirements, control measures, documentation, testing, and consequences of non-compliance related to information security under SOX to effectively contribute to their organization's compliance efforts. ## **Background** SOX, officially known as the Sarbanes-Oxley Act of 2002, was enacted in response to corporate scandals like Enron and WorldCom, which undermined investor confidence. It is a U.S. federal law that sets requirements for financial reporting and corporate governance. ## **Section 404** This is a crucial section of SOX for security professionals. It requires companies to establish and maintain adequate internal controls over financial reporting systems. In other words, it mandates that companies have security measures in place to ensure the accuracy and integrity of financial data. Security software engineers should understand the importance of data integrity in financial reporting. ## **Control Measures** Engineers should be familiar with the types of control measures that can help achieve compliance with Section 404. This includes implementing access controls, encryption, audit trails, and other security mechanisms to protect financial data from unauthorized access, tampering, or fraud. ## **Documentation** SOX compliance involves thorough documentation of security policies, procedures, and controls. Engineers should know how to create and maintain clear and comprehensive documentation to demonstrate compliance. ## **Testing and Auditing** Section 404 requires regular testing and auditing of internal controls to ensure they are effective. Security engineers should be prepared to assist in these efforts, helping to identify weaknesses and vulnerabilities in security controls. ## **Penalties for Non-Compliance** Engineers should understand the potential consequences of non-compliance with SOX. Failure to meet the requirements can result in financial penalties, legal liabilities, and damage to a company's reputation. ## **Impact on IT Systems** Engineers need to consider how SOX compliance affects IT systems, especially those involved in financial reporting. They should ensure that security measures are integrated into these systems to meet SOX requirements. ## **Ongoing Compliance** SOX compliance is an ongoing process. Engineers should be aware that maintaining compliance requires continuous monitoring, assessment, and improvement of security controls. # HIPAA and HITECH In summary, security software engineers should be well-versed in the HIPAA Security Rule, the breach notification requirements of the HITECH Act, and the broader security best practices for protecting electronic personal health information (ePHI). This knowledge is critical for designing, developing, and maintaining healthcare-related software systems that handle sensitive patient information securely and in compliance with these important healthcare regulations. ## **Purpose** HIPAA and HITECH are U.S. federal laws that regulate the privacy and security of protected health information (PHI) and promote the adoption of electronic health records (EHRs). ### **HIPAA** #### **Security Rule** Engineers should be familiar with the HIPAA Security Rule, which sets standards for safeguarding electronic PHI (ePHI). This includes requirements for access controls, encryption, audit logging, and other security measures. #### **Risk Assessment** Engineers should understand the importance of conducting regular risk assessments to identify and mitigate security risks to ePHI. This is a fundamental requirement under HIPAA. #### **Business Associate Agreements** HIPAA mandates that covered entities (e.g., healthcare providers) have business associate agreements in place with third parties (e.g., software vendors) who handle ePHI. Engineers should know the security obligations these agreements entail. ### **HITECH Act** #### **Breach Notification** Engineers should be aware of the HITECH Act's breach notification requirements, which mandate that covered entities report breaches of unsecured PHI to affected individuals, the Department of Health and Human Services (HHS), and, in some cases, the media. #### **Increased Penalties** HITECH introduced higher penalties for non-compliance with HIPAA, making it essential for engineers to help ensure security measures are in place to avoid these penalties. ### **Security Best Practices** * **Access Control:** Engineers should understand the importance of implementing robust access controls to limit access to ePHI based on the principle of least privilege. * **Data Encryption:** They should know the significance of encrypting ePHI both in transit and at rest to protect it from unauthorized access. * **Audit Logging:** Engineers should be familiar with audit logging requirements to track and monitor access to ePHI, helping in identifying and responding to security incidents. * **Incident Response:** Engineers should be prepared to assist in developing and implementing incident response plans to address security breaches or incidents involving ePHI. * **Training and Awareness:** HIPAA and HITECH emphasize the need for workforce training and awareness programs to educate employees about security policies and procedures. Engineers can contribute to these efforts. * **Ongoing Compliance:** Compliance with HIPAA and HITECH is an ongoing process. Engineers should understand the need for continuous monitoring, periodic risk assessments, and updates to security measures to maintain compliance. # Payment Card Industry Data Security Standard (PCI-DSS) In summary, a security software engineer should be well-versed in the PCI DSS standards, including the specific security requirements, best practices for secure software development, incident response, and the compliance and validation processes. This knowledge is vital for designing, developing, and maintaining software systems that handle payment card data securely and in compliance with PCI DSS standards. ## **PCI DSS Overview** **Purpose:** Payment Card Industry (PCI) Data Security Standard (DSS) is a set of security standards established to protect payment card data, including credit card and debit card information, during processing, transmission, and storage. **Scope:** Engineers should recognize that PCI DSS applies to any organization that accepts, processes, stores, or transmits payment card data. This includes retailers, e-commerce sites, payment processors, and financial institutions. ## **Key Security Requirements** **Data Encryption:** Engineers should understand the importance of encrypting cardholder data both in transit and at rest. They should be familiar with encryption standards and protocols. **Access Control:** Knowledge of access control mechanisms, such as role-based access and least privilege principles, is essential for ensuring that only authorized individuals can access payment card data. **Network Security:** Engineers should be aware of network segmentation and firewall requirements to protect cardholder data environments (CDE) from unauthorized access. **Vulnerability Management:** Understanding vulnerability assessment and patch management is crucial for addressing security vulnerabilities promptly. ## **Secure Software Development** **Secure Coding Practices:** Engineers should follow secure coding practices to develop applications that handle payment card data securely. This includes input validation, secure authentication, and secure error handling. **Change Management:** Knowledge of change control processes and the impact of code changes on security is important to maintain PCI DSS compliance. ## **Incident Response** **Incident Handling:** Engineers should be prepared to assist in developing and implementing an incident response plan to address security incidents involving payment card data. ## **Logging and Monitoring** **Audit Trails:** Engineers should understand the importance of audit trails and logging to monitor and detect suspicious activities related to payment card data. **Security Information and Event Management (SIEM):** Familiarity with SIEM tools and practices is crucial for real-time monitoring and alerting. ## **Compliance and Validation** **Self-Assessment Questionnaire (SAQ):** Engineers should know about SAQs, which are used to assess compliance for organizations that do not require a full-scale audit. **Annual Assessments:** Engineers should be aware that organizations must undergo annual PCI DSS assessments, either through self-assessment or a Qualified Security Assessor (QSA) for larger entities. ## **Penalties and Liabilities** **Non-Compliance:** Engineers should understand the consequences of non-compliance with PCI DSS, which can include fines, restrictions, or termination of the ability to process payment card transactions. # General Data Protection Regulation (GDPR) Compliance with GDPR requires a holistic approach to data protection and security. Security software engineers should collaborate with legal and compliance experts to ensure that software systems align with GDPR requirements, as non-compliance can result in significant fines and reputational damage. ## **Scope of GDPR** Understand that GDPR applies to organizations, including software providers, that process personal data of individuals within the EU, regardless of where the organization is based. Personal data is defined as any information relating to an identified or identifiable natural person. This includes: * Online identifiers * IP addresses * Cookies It also includes indirect information, including physical, physiological, genetic, mental, economic, cultural, or social identities that can be traced to one person. ## Obligations GDPR mandates that individuals have the right to access clear and comprehensible information regarding the processing of their data. When organizations collect data from individuals, they are obligated to transparently convey key details, including: * The identity and contact information of the requesting entity * The purpose and usage of the data * The retention period * Potential international transfers * **The individual's rights such as access, rectification, erasure (right to be forgotten), withdrawal of consent, and the ability to file complaints.** # California Consumer Privacy Act (CCPA) This is California's bill that is similar to the GDPR. Under the CCPA, consumers should have the following rights: * Possess the right to understand the purpose of personal data usage. * Possess the right to be informed about and object to data sales. * Possess the right to know the recipients of their data. * Be free from discrimination for objecting to data sales. * Possess the right to access their held data. # Federal Risk and Authorization Management Program (FedRAMP) FedRAMP, short for the Federal Risk and Authorization Management Program, is a U.S. government program established to standardize the security assessment, authorization, and continuous monitoring processes for cloud products and services. Its primary goal is to ensure that cloud solutions used by federal agencies meet consistent and stringent security standards. Based on **NIST Special Publication 800-53**, it defines security controls and families. Controls address aspects like Access Control, Communications Protection, and Security Assessment. ### FedRAMP Marketplace The FedRAMP Marketplace is an online repository that provides a comprehensive list of cloud products and services that have received authorization through the Federal Risk and Authorization Management Program (FedRAMP). It serves as a valuable resource for federal agencies and organizations seeking FedRAMP-compliant cloud solutions. Examples of FedRAMP Authorized Cloud Services: * **Amazon Web Services (AWS)** * **Microsoft Azure** * **Google Cloud Platform (GCP)** * **Salesforce Government Cloud** * **ServiceNow** * **Adobe Document Cloud for Government** * **Oracle Cloud Infrastructure (OCI)** ### # Privacy & Data Privacy refers to an individual's right to keep their personal information, activities, and communications confidential. ## Personally Identifiable Information (PII) This refers to information that can be used to identify someone. * Full name * National identification number (i.e., SSN) * IP address * Home address * Motor vehicle registration plate number * Driver’s license or state ID number * Face, fingerprints, or handwriting * Credit card and bank account numbers * Date of birth * Birthplace * Genetic information ## Data Data refers to raw and unprocessed facts, figures, symbols, or information that can be in the form of numbers, text, images, or any other format. Data can be collected, stored, and manipulated to extract meaningful insights, support decision-making, or perform various tasks. ### Data Classification Data classification is a multifaceted process, encompassing state, use, and security importance, serving as a risk management tool to align protection costs with asset value and potentially becoming more intricate for larger enterprises with diverse data protection requirements, including compliance considerations. #### Data States * At Rest/Being Stored * Being Created/Generated * Being Transmitted * Being Updated/Deleted #### Data Usage * Input * Output * Initialized (internal) #### Data Risk Impact | Impact | Damage | Financial Consequence | | ------ | ------------------------------------- | ----------------------- | | High | Death | Greater than $1,000,000 | | Medium | Severe Injury / Loss of Functionality | Greater than $100,000 | | Low | Minor Injury | Less than $100,000 | # Introduction to Linux Here are some core concepts behind Linux that are important to know about: ## Concepts * Kernel * Distribution * Boot Loader * Service * Filesystem * X Window System * Desktop Environment * Command Line ## Kernel **Think of it as the brain of your computer.** It manages hardware, system resources, and communication between software and hardware. <div align="left" data-full-width="false"><figure><img src="/files/s741YJ9ES6vIDuhD8NCm" alt="" width="188"><figcaption></figcaption></figure></div> ## Distribution A version of Linux that includes **the linux kernel plus additional software, tools, and package management.** It's like a customized Linux operating system, such as Ubuntu or Fedora. ## Boot Loader A small program that runs first when you start your computer, loading the operating system (like GRUB). It's the gatekeeper to the OS. ## Service A program or process that runs in the background to provide functionality (e.g., web server, database). Services keep the system running smoothly without direct interaction. ## Filsystem The way data is stored and organized on your computer. It's like the file cabinet of your computer where everything is saved and accessed. ## X Window System The system that handles the graphical display on your screen (windows, icons, etc.) and enables interaction with input devices (like the mouse and keyboard). It's **the foundation** for your graphical user interface (GUI), but **doesn't include the look and feel of the interface itself**. ## Desktop Environment The visual user interface that lets you interact with your computer (e.g., GNOME, KDE). It includes icons, windows, and apps, making your system **visually functional**. ## Command Line A text-based interface to interact with the system by typing commands. It's a powerful, faster way to control your computer compared to clicking around. # Architecture <img src="/files/NZdNI3qwEgu8ZASgaHq2" alt="" data-size="original"> * The Linux kernel is monolithic in nature. * System calls are used to interact with the Linux kernel space. * Kernel code can only be executed in the kernel mode. Non-kernel code is executed in the user mode. * Device drivers are used to communicate with the hardware devices. # Server Administration ### <a href="#multi-user-operating-systems" id="multi-user-operating-systems"></a> # User / Groups ### Multi-User Operating Systems <a href="#multi-user-operating-systems" id="multi-user-operating-systems"></a> An operating system is considered as multi-user if it allows multiple people/users to use a computer and not affect each other's files and preferences. Linux based operating systems are multi-user in nature. ### User/Group Management <a href="#usergroup-management" id="usergroup-management"></a> * Users in Linux has an associated user ID called UID attached to them. * A group is a collection of one or more users. * A group makes it easier to share permissions among a group of users. * Each group has a group ID called GID associated with it. #### id command <a href="#id-command" id="id-command"></a> `id` command can be used to find the uid and gid associated with an user. It also lists down the groups to which the user belongs to. The uid and gid associated with the root user is 0. ![](/files/tI0F78FG7bvp4sUOppSi) A good way to find out the current user in Linux is to use the whoami command. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image35.png) **"root" user or superuser is the most privileged user with** **unrestricted access to all the resources on the system. It has UID 0** Important files associated with users/groups | /etc/passwd | Stores the user name, the uid, the gid, the home directory, the login shell etc | | ----------- | ---------------------------------------------------------------------------------- | | /etc/shadow | Stores the password associated with the users. Can only be accessed by super users | | /etc/group | Stores information about different groups on the system | ### Important commands for managing users <a href="#important-commands-for-managing-users" id="important-commands-for-managing-users"></a> Some of the commands which are used frequently to manage users/groups on Linux are following: * `useradd` - Creates a new user * `passwd` - Adds or modifies passwords for a user * `usermod` - Modifies attributes of a user (like home directory or shell) * One easy way of providing root access to users is to add them to a group which has permissions to run all the commands. "wheel" is a group in redhat Linux with such privileges. `usermod -a -G wheel <user>` * `userdel` - Deletes a user * `su` - Switch user ### Important commands for managing groups <a href="#important-commands-for-managing-groups" id="important-commands-for-managing-groups"></a> | groupadd \\\<group\_name> | Creates a new group | | ------------------------- | ------------------------------ | | groupmod \\\<group\_name> | Modifies attributes of a group | | groupdel \\\<group\_name> | Deletes a group | | gpasswd \\\<group\_name> | Modifies password for group | # File Permissions On a Linux operating system, each file and directory is assigned access permissions for the owner of the file, the members of a group of related users and everybody else. This is to make sure that one user is not allowed to access the files and resources of another user. ![](/files/6X889or3XOfF5cfoPkj4) ![](/files/d2XVPPX7K6w4CNXeVEpX) ![](/files/QK22VFL8BcJ9Xaw6Z0ow) #### Chmod command <a href="#chmod-command" id="chmod-command"></a> The chmod command is used to modify files and directories permissions in Linux. The chmod command accepts permissions in as a numerical argument. We can think of permission as a series of bits with 1 representing True or allowed and 0 representing False or not allowed. | Permission | rwx | Binary | Decimal | | ----------------------- | --- | ------ | ------- | | Read, write and execute | rwx | 111 | 7 | | Read and write | rw- | 110 | 6 | | Read and execute | r-x | 101 | 5 | | Read only | r-- | 100 | 4 | | Write and execute | -wx | 011 | 3 | | Write only | -w- | 010 | 2 | | Execute only | --x | 001 | 1 | | None | --- | 000 | 0 | Each digit is independent of the other two. Therefore, `750` means the current user can read, write, and execute, the group cannot write, and others cannot read, write, or execute. `744`, which is a typical default permission, allows read, write, and execute permissions for the owner, and read permissions for the group and “world” users. #### `664 => rw-rw-r--` <a href="#chmod-664-rw-rw-r" id="chmod-664-rw-rw-r"></a> ``` chmod 664 example.txt ``` \ **Chown command** The chown command is used to change the owner of files or directories in Linux. Command syntax: chown \\\<new\_owner> \\\<file\_name> ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image6.png) **Chgrp command** The chgrp command can be used to change the group ownership of files or directories in Linux. The syntax is very similar to that of chown command. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image27.png) Chgrp command can also be used to change the owner of a directory in the similar way.\ # SSH The ssh command is used for logging into the remote systems, transfer files between systems and for executing commands on a remote machine. SSH stands for secure shell and is used to provide an encrypted secured connection between two hosts over an insecure network like the internet. Reference: <https://www.ssh.com/ssh/command/> #### Passwordless Authentication Using SSH <a href="#passwordless-authentication-using-ssh" id="passwordless-authentication-using-ssh"></a> Using this method, we can ssh into hosts without entering the password. This method is also useful when we want some scripts to perform ssh-related tasks. Steps for setting up a passwordless authentication with a remote host: 1. Install openssh package which contains all the commands related to ssh. 2. Generate a key pair using the ssh-keygen command. 3. After running the ssh-keygen command successfully, we should see two keys present in the `~/.ssh` directory. `id_rsa` is the private key and `id_rsa.pub` is the public key. Do note that the private key can only be read and modified by you. 4. Transfer the public key to the remote host 1. There are multiple ways to transfer the public key to the remote server. We will look at one of the most common ways of doing it using the ssh-copy-id command. 2. Install the openssh-clients package to use ssh-copy-id command. 5. Our public key should be in `~/.ssh/authorized_keys` now. `~/.ssh/authorized_key` contains a list of public keys. The users associated with these public keys have the ssh access into the remote host. #### How to run commands on a remote host ? <a href="#how-to-run-commands-on-a-remote-host" id="how-to-run-commands-on-a-remote-host"></a> General syntax: ssh \\\<user>@\\\<hostname/hostip> \\\<command> #### How to transfer files from one host to another host ? <a href="#how-to-transfer-files-from-one-host-to-another-host" id="how-to-transfer-files-from-one-host-to-another-host"></a> General syntax: scp \\\<source> \\\<destination> # Process Management Some useful commands that can be used to monitor the processes on Linux systems. #### ps (process status) <a href="#ps-process-status" id="ps-process-status"></a> The ps command is used to know the information of a process or list of processes. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image24.png) If you get an error "ps command not found" while running ps command, do install **procps** package. ps without any arguments is not very useful. Let's try to list all the processes on the system by using the below command. Reference: <https://unix.stackexchange.com/questions/106847/what-does-aux-mean-in-ps-aux> ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image42.png) We can use an additional argument with ps command to list the information about the process with a specific process ID. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image2.png) We can use grep in combination with ps command to list only specific processes. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image1.png) **top** The top command is used to show information about Linux processes running on the system in real time. It also shows a summary of the system information. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image53.png) For each process, top lists down the process ID, owner, priority, state, cpu utilization, memory utilization and much more information. It also lists down the memory utilization and cpu utilization of the system as a whole along with system uptime and cpu load average.\ \ **Memory Management** In this section, we will study about some useful commands that can be used to view information about the system memory. #### free <a href="#free" id="free"></a> The free command is used to display the memory usage of the system. The command displays the total free and used space available in the RAM along with space occupied by the caches/buffers. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image22.png) free command by default shows the memory usage in kilobytes. We can use an additional argument to get the data in human-readable format. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image5.png) Checking Disk Space In this section, we will study about some useful commands that can be used to view disk space on Linux. #### df (disk free) <a href="#df-disk-free" id="df-disk-free"></a> The df command is used to display the free and available space for each mounted file system. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image36.png) #### du (disk usage) <a href="#du-disk-usage" id="du-disk-usage"></a> The du command is used to display disk usage of files and directories on the system. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image10.png) The below command can be used to display the top 5 largest directories in the root directory. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image18.png) Systemd Systemd is a system and service manager for Linux operating systems. Systemd units are the building blocks of systemd. These units are represented by unit configuration files. The below examples shows the unit configuration files available at /usr/lib/systemd/system which are distributed by installed RPM packages. We are more interested in the configuration file that ends with service as these are service units. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image16.png) #### Managing System Services <a href="#managing-system-services" id="managing-system-services"></a> Service units end with .service file extension. Systemctl command can be used to start/stop/restart the services managed by systemd. | Command | Description | | ------------------------------ | ------------------------------------- | | systemctl start name.service | Starts a service | | systemctl stop name.service | Stops a service | | systemctl restart name.service | Restarts a service | | systemctl status name.service | Check the status of a service | | systemctl reload name.service | Reload the configuration of a service | ### Logs <a href="#logs" id="logs"></a> In this section, we will talk about some important files and directories which can be very useful for viewing system logs and applications logs in Linux. These logs can be very useful when you are troubleshooting on the system. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image58.png) \ \ \ # Networking # Diagrams ![](/files/oQbfv4lajOudQZ7fFe96) <img src="/files/HJeZu28CE8U7jHTmmvyX" alt="" data-size="original"> {% embed url="<https://media.licdn.com/dms/image/v2/D4D22AQHpYiUURLgI9g/feedshare-shrink_800/feedshare-shrink_800/0/1703939855448?e=1730332800&t=OVKffRHldUStGpGG61a-FDtse2wPXZ2VTTRWGyf99kM&v=beta>" %} # Browser URL Example What happens when you start typing google in your browser's search bar? <https://github.com/alex/what-happens-when> ### Simplified #### Keyboard Actions * **Key Pressed**: "g" key pressed; browser processes it and suggests URLs based on history. * **Enter Key**: Physical keypress leads to electrical signals transmitted to the computer. * USB keyboard: Current flows to logic circuitry, keycode is generated and sent. * Virtual keyboard: Capacitive touch registers keypress, sends a software interrupt. #### OS-Level Events * **Interrupts**: * OS detects the keypress and triggers an interrupt request. * **Windows**: Sends a `WM_KEYDOWN` message. * **macOS**: Sends an `NSEvent`. * **Linux**: Xorg server listens for keycodes. #### URL Parsing * **URL or Search Term**: Determines if the input is a URL or search term. * **Non-ASCII Characters**: Converts characters using Punycode if needed. #### Network Communication * **DNS Lookup**: * If the browser doesn't find the domain in its local cache, it sends a request to the DNS server (usually provided by the ISP or a local router). * If the DNS server doesn't have the answer, it will act as a recursive resolver, querying other DNS servers higher up in the hierarchy on behalf of the client. <figure><img src="/files/CFIyknl7GSIjhhRyJteM" alt=""><figcaption></figcaption></figure> * **ARP Process**: Determines the MAC address for the DNS server or default gateway. * **Socket Opening**: Establishes a connection to the server using TCP. * **TLS Handshake**: Encrypts communication using SSL/TLS for HTTPS. #### HTTP Protocol * **Request**: Sends an HTTP request to the server (e.g., GET request). * **Response**: Server returns HTML, possibly with a status code (e.g., `200 OK`). * **Resources**: Browser fetches additional resources (images, CSS) as needed. #### Browser Rendering * **HTML Parsing**: Parses the HTML document into a DOM tree. * **CSS Parsing**: Interprets CSS to apply styles. * **Rendering**: Constructs a render tree, lays out elements, and paints them on the screen. * **GPU Rendering**: Uses GPU for faster rendering in modern browsers. #### Post-Rendering * **JavaScript Execution**: Runs any JavaScript on the page. * **User Interaction**: Scripts and network requests can be triggered by user input. # Network Topologies Notes taken from: https\://explained-from-first-principles.com/internet/#nodes-and-links ### Nodes and links <a href="#nodes-and-links" id="nodes-and-links"></a> Nodes (computers) communicate with eachother over links (channels). In graph theory, nodes are called vertices and links are called edges. <figure><img src="/files/BC8ZvsFWda9mshEu0lZn" alt=""><figcaption>Nodes connected by a Link</figcaption></figure> ### Fully Connected Network A fully connected network has direct links from every node to every other node. This scales badly in networking as the number of links grows quadratically with the number of nodes. The formula is `l = n x (n - 1) / 2` In graph theory, this is called a complete graph. <figure><img src="/files/Byu8J7Z9ziwrWerHhdNP" alt=""><figcaption></figcaption></figure> ### Star Network Introducing a central node (**router / relaying node**) reduces the number of links between nodes. In a star network, the number of links scales linearly with the number of nodes. Non-relaying nodes act as communication endpoints. **A star network is centralized**, with the central node exerting control over communication and being a single point of failure. This topology **reduces availability** and may be undesirable for political and technical reasons. <figure><img src="/files/NBOUAPZ4ysLiZ6H1tcMy" alt=""><figcaption>Star network with central node (router) in blue</figcaption></figure> ### Mesh Network Increasing the number of routers (relaying nodes) avoids the drawbacks of centralization. In a mesh network, communication remains possible even if some routers fail. A partially connected network **balances redundancy and scalability.** Mesh networks are typically **preferred for flexibility and reliability**. Critical systems may connect to multiple routers for **increased availability, despite higher costs**. <figure><img src="/files/KNrkIFdMIqzu2JjZwHKk" alt=""><figcaption>Mesh Network with multiple routers</figcaption></figure> # Signal Routing ### **Network Addresses** In networks with routers, nodes need a way to address each other. Even if a router forwards signals to all its links like a hub, nodes must determine if they are the intended recipients of messages. This is achieved by assigning a unique identifier, called a **network address**, to each node. **Each message includes the recipient's identifier, allowing routers to know where to forward communications.** This system works best when addresses are assigned based on the network's geographical structure rather than randomly. <figure><img src="/files/pqETJYVBVxRsnfUgkyJP" alt=""><figcaption>Nodes with addresses according to the router they’re connected to</figcaption></figure> ### **Routing Tables** Routing is the process of selecting a path between two nodes across a network. Routers handle this task using a routing table, which tells the router where to forward communication for each node. For example, in the routing table below, router A forwards messages for node A2 on link 2, regardless of where it receives them from. **Example Routing Table for Router A:** | Destination | Link | Cost | | ----------- | ---- | ---- | | A1 | 1 | 4 | | A2 | 2 | 2 | | B? | 3 | 5 | | B? | 4 | 8 | | C? | 3 | 9 | | C? | 4 | 6 | The router chooses routes based on cost, which could be based on network delay or transmission cost. In this case, router A forwards communications for nodes starting with C on link 4 because it has a lower cost. ### **Forwarding Tables** A forwarding table is a simplified version of the routing table. It only contains the optimal routes, which makes it smaller and faster for lookup, important for low-latency routing. **Example Forwarding Table for Router A:** | Destination | Link | | ----------- | ---- | | A1 | 1 | | A2 | 2 | | B? | 3 | | C? | 4 | ### **Routing Protocols** Routers can fail, and new nodes are added to networks often. To keep routing tables current, routers use routing protocols to communicate changes. For instance, if router A stops hearing from router C, it will update its table to send communications to C through router B. ### **Signal Relaying** Signals can be relayed through a network using either circuit switching or packet switching. ### **Circuit Switching** In a circuit-switched network, a dedicated communication channel is established for the duration of the call. * **Example**: Early telephone networks required a switchboard operator to connect two telephones, creating a closed circuit. * **Advantages**: Signal delay remains constant, and messages arrive in the same order. * **Disadvantages**: Inefficient use of capacity, as other users cannot access the channel when it's idle (e.g., during pauses in conversation). ### **Packet Switching** In a packet-switched network, data is divided into packets, which consist of a header and a payload. * **Header**: Contains delivery information, such as sender and recipient addresses. * Each router has a queue for incoming packets and forwards them based on its forwarding table. * Packet-switching routers do not maintain state, meaning no channels are opened or closed at the routing level. # DNS (Domain Name System) ### **What is DNS?** The Domain Name System (DNS) is a decentralized naming system for devices and services connected to the internet or private networks. It translates human-friendly domain names (like google.com) into machine-friendly IP addresses (such as 192.168.0.1). This allows browsers to locate and load resources on the internet without needing to memorize numerical IP addresses. ### **What layer of the OSI model does it operate at?** **DNS (Domain Name System)** operates at the **Application Layer** (Layer 7) of the OSI model. ### **How Does DNS Work?** DNS resolves human-readable domain names into IP addresses. This process involves four main components: 1. **DNS Recursor**: A server acting as a "librarian" that receives queries from browsers and initiates additional requests to fulfill the DNS query. 2. **Root Nameserver**: The first step in finding an IP address, similar to a library index, directing the query to more specific servers. 3. **TLD Nameserver**: Points to servers associated with the top-level domain (like .com or .net). 4. **Authoritative Nameserver**: The final stop that stores the actual IP address for the requested domain. ### **DNS Lookup Steps:** <figure><img src="/files/QueGXmUbIe65cDCgD8Jo" alt=""><figcaption><a href="https://www.iotforall.com/a-simple-explanation-of-the-domain-name-system">https://www.iotforall.com/a-simple-explanation-of-the-domain-name-system</a></figcaption></figure> 1. A user types in a domain like "example.com." 2. The browser queries a **DNS resolver**. 3. The recursor asks the **root nameserver**, which points to the relevant **TLD server**. 4. The **TLD server** provides the IP address of the domain’s nameserver. 5. The recursor gets the IP address from the authoritative nameserver and returns it to the browser. The browser can now load the web page by sending a request to that IP. Here's another image showing the step by step process: <figure><img src="/files/7QQmaUfePMZoeMecDXfA" alt=""><figcaption><a href="https://bluecatnetworks.com/glossary/what-is-a-dns-server/">https://bluecatnetworks.com/glossary/what-is-a-dns-server/</a></figcaption></figure> ### **Types of DNS Queries:** 1. **Recursive Query**: The DNS client expects a definitive answer (either the record or an error). 2. **Iterative Query**: The DNS server returns the best possible answer or a referral to another server. 3. **Non-Recursive Query**: This occurs when the DNS resolver already has the information cached and returns it without needing further queries. ### **DNS Caching:** To speed up the process, DNS records are cached in various locations: * **Browser Cache**: Modern browsers store DNS records locally for a short time. * **Operating System Cache**: The OS also caches DNS records to avoid repetitive queries. * **ISP Caching**: ISPs store frequently requested DNS records to reduce query time further. In some cases, Cloudflare DNS optimizes this process with infrastructure designed to handle high traffic, such as root DNS servers. # SSL (Secure Sockets Layer) <figure><img src="/files/JqapGMHgxF6hLsEWLblH" alt=""><figcaption><a href="https://www.cloudflare.com/en-ca/learning/ssl/what-is-ssl/">https://www.cloudflare.com/en-ca/learning/ssl/what-is-ssl/</a></figcaption></figure> ## SSL (Secure Sockets Layer) Overview * SSL is an encryption-based security protocol developed by Netscape in 1995. * Ensures privacy, authentication, and data integrity in online communications. * SSL is the predecessor to **TLS (Transport Layer Security)**, the modern encryption standard. ## How SSL/TLS Works 1. **Data Encryption**: * SSL encrypts data transmitted over the internet, making intercepted data unreadable. 2. **Authentication**: * SSL initiates a **handshake** process to verify the identity of both communicating devices. 3. **Data Integrity**: * SSL digitally signs data, ensuring that it hasn’t been tampered with during transmission. ## Importance of SSL/TLS * Originally, data was transmitted in **plaintext**, vulnerable to interception. * SSL encrypts sensitive information (e.g., credit card numbers) to prevent theft. * Authenticates websites to avoid **phishing** attacks and ensures data isn’t altered. ## SSL vs TLS * **TLS** (Transport Layer Security) is the updated version of SSL, introduced in 1999. * SSL 3.0 (last version of SSL) is deprecated due to known vulnerabilities. * Most websites now use TLS, although many still refer to it as **SSL**. ## SSL Certificates * Websites must have an **SSL certificate** (technically a TLS certificate) to implement SSL/TLS encryption. * **Public key** encryption is used for establishing secure communication. * SSL certificates are issued by trusted **Certificate Authorities (CAs)**. ### Types of SSL Certificates * **Single-domain**: Covers one domain (e.g., [www.example.com](http://www.example.com/)). * **Wildcard**: Covers one domain and all its subdomains (e.g., [www.example.com](http://www.example.com/), blog.example.com). * **Multi-domain**: Covers multiple unrelated domains. ### SSL Certificate Validation Levels 1. **Domain Validation (DV)**: * Basic level; verifies control of the domain. 2. **Organization Validation (OV)**: * More involved; CA directly contacts the organization. 3. **Extended Validation (EV)**: * Highest level; requires a full background check of the organization. ## Obtaining an SSL Certificate * **Cloudflare** offers free SSL certificates with easy setup. * Some websites may need additional setup for their origin servers. # TLS (Transport Layer Security) ## Transport Layer Security (TLS) Overview * **TLS** is a widely used security protocol designed for ensuring privacy and data security in online communications. * Primarily encrypts communication between web applications and servers (e.g., web browsers and websites). * Other uses: Encrypting emails, messaging, and VoIP. * **First introduced** in 1999 by the **Internet Engineering Task Force (IETF)**; latest version is **TLS 1.3** (published in 2018). ## TLS vs SSL * **TLS** evolved from the earlier **SSL** protocol (Secure Sockets Layer), developed by Netscape. * **TLS 1.0** started as **SSL 3.1**, but was renamed before release to disassociate from Netscape. * **TLS** and **SSL** are often used interchangeably due to their close relationship. ## TLS vs HTTPS * **HTTPS** is HTTP combined with **TLS encryption**. * Any website using HTTPS is utilizing **TLS** to encrypt communication. ## Why Should Businesses Use TLS? * **TLS encryption** protects against data breaches and cyberattacks. * Major browsers (e.g., **Google Chrome**) warn users about non-HTTPS sites. * **HTTPS padlock icon** is a sign of a secure, TLS-protected connection. ## TLS Components 1. **Encryption**: Hides transferred data from third parties. 2. **Authentication**: Verifies the identity of communicating parties. 3. **Integrity**: Ensures data has not been altered during transmission. ## TLS Certificates * To use TLS, a website must have a **TLS certificate** (commonly referred to as an **SSL certificate**). * Issued by **certificate authorities (CAs)** to the domain owner. * Contains the domain’s ownership details and **public key** (used for encryption). ## How TLS Works #### TLS Handshake * The handshake process establishes a secure connection between the user's device (client) and the web server. 1. **Specify TLS version**: (e.g., TLS 1.0, 1.2, 1.3). 2. **Choose cipher suites**: Set of algorithms for encryption. 3. **Server authentication**: Verifies server’s identity using the **TLS certificate**. 4. **Generate session keys**: For encrypting messages after the handshake. * **Public key cryptography**: Uses a public key to decrypt data, while only the server’s private key can encrypt it. * After encryption and authentication, the data is signed with a **Message Authentication Code (MAC)** for integrity. ## Performance Impact of TLS * Modern **TLS versions** (e.g., **TLS 1.3**) have minimal impact on web performance. * **TLS False Start** and **Session Resumption** mitigate potential latency by speeding up the handshake. * **TLS 1.3** improves speed with a 1-round-trip handshake and zero round trips for previously connected users. ## Implementing TLS * **Cloudflare** offers free TLS/SSL certificates. * Alternatively, businesses can acquire and install an **SSL certificate** from a **certificate authority** on their servers. # Process ## Requirements #### **Key Simplified Workflow** 1. Write 3–5 **Functional Requirements**. 2. Write 3–5 **Non-Functional Requirements** with specific metrics. 3. Skip detailed capacity math unless it's crucial for design decisions. *** ### **Functional Requirements (Core Features)** * **Goal**: Identify the top 3 features your system must have. * **Approach**: * Think about what **users** or **clients** will do with the system. * Use **"Should be able to..."** statements to frame these features. * Example: * For Twitter: "Users should be able to post tweets." * For a Cache: "Clients should be able to read items." 💡 **Tip**: Prioritize only the most critical features. Skip minor details or edge cases. *** ### **Non-Functional Requirements (Qualities)** * **Goal**: Define key qualities your system should meet (e.g., speed, reliability). * **Approach**: * Use **"The system should..."** statements, with specific metrics when possible. * Example: * For Twitter: "The system should render feeds in under 200ms." * For a Cache: "The system should tolerate one node failure without data loss." 💡 **Checklist for Non-Functional Needs**: * **CAP Theorem**: Consistency or Availability? * **Scalability**: Will traffic grow, and how (e.g., bursty traffic)? * **Latency**: Which operations need to be fast (and how fast)? * **Durability**: Is data loss acceptable? (Critical for banking; less so for logs.) * **Fault Tolerance**: How many failures should the system survive? *** ### **Capacity Estimation (Optional)** * Skip this unless **calculations affect your design**. * Example: * For TopK trending topics: Estimate the number of topics to decide between using a single machine or sharding data. 💡 **Tip**: Explain to the interviewer that you'll do math only if necessary, focusing instead on relevant design trade-offs. *** ## **Core Entities** **Goal**: Identify the main objects in your system to build your foundation. * **Approach**: * **Who are the actors?** (e.g., users, clients). * **What nouns/resources fulfill functional requirements?** * Jot down a **bulleted list** of key entities. This is a **first draft**, so avoid overcomplicating with details like relationships or fields. * **Example (Twitter)**: * **User** * **Tweet** * **Follow** 💡 **Tip**: Use clear, intuitive names for entities to make it easier for everyone to follow. *** ## **API or System Interface** **Goal**: Define how the system communicates with users or clients (its **contract**). * **Decision**: Choose a communication protocol: * **REST API** (default for simplicity): Uses standard HTTP verbs (GET, POST, etc.). * **GraphQL**: Use only if clients need fine-grained control over the data returned. * **Wire Protocol**: Define a message format for WebSocket or TCP connections. * **Steps**: * Base your endpoints on **core entities**. * Write endpoint definitions that map to **functional requirements**. * **Example (Twitter REST API)**: * **POST /v1/tweet** * Body: `{ "text": string }` * Authenticated user ID inferred from token. * **GET /v1/tweet/:tweetId** → Returns a **Tweet**. * **POST /v1/follow/:userId** → Authenticated user follows another user. * **GET /v1/feed** → Returns a list of **Tweet\[]**. 💡 **Security Tip**: Avoid putting sensitive data like `userId` in the body or query params when it can be derived securely from headers (e.g., auth tokens). *** ## **High-Level Design** **Goal**: Build a simple architecture to meet your system’s functional requirements and API design. * **Focus**: * **Components**: Servers, databases, caches, message queues, etc. * **Interaction**: Draw arrows to show how components communicate. * **Keep It Simple**: Start with basic components and focus on functional requirements. * **Step-by-Step**: 1. **Identify core components**: What do you need for your system to work? For example, a web server, a database, etc. 2. **Flow of data**: Discuss how data flows from one component to another, focusing on state changes during each request. 3. **Start with API endpoints**: Build your design around these endpoints, adding complexity only when necessary (e.g., caching, queuing). * **Visual Tips**: * You don’t need to document every single detail in your schema. Focus on **important fields** that affect your design. * As you draw your architecture, talk through your thought process, explaining how each component interacts. * **Example (Twitter)**: * POST `/v1/tweet` → Send to server → Store in database. * GET `/v1/tweet/:tweetId` → Fetch from database → Return to client. * POST `/v1/follow/:userId` → Update user’s following list → Store in database. * GET `/v1/feed` → Use cached feeds or query database for timeline. *** ## **Deep Dives** **Goal**: Harden your design by addressing non-functional requirements, edge cases, and bottlenecks. * **Focus**: 1. **Non-functional requirements**: Scalability, performance, fault tolerance. 2. **Edge cases**: Handle unexpected conditions like failed API requests or database downtime. 3. **Bottlenecks**: Identify and improve performance issues. 4. **Probes**: Address any interviewer feedback or questions during this section. * **Proactive Discussion**: * **Junior candidates**: Your interviewer will guide you, pointing out places to improve. * **Senior candidates**: Lead the discussion by identifying potential problems and discussing solutions. * **Example (Twitter)**: * **Scalability**: Horizontal scaling, caching, and database sharding to handle >100M DAU. * **Feed Latency**: Discuss **fanout-on-read** vs **fanout-on-write** strategies and caching to minimize latency when fetching user feeds. * **Collaboration Tip**: Don’t dominate the conversation. Allow space for the interviewer to probe your design. Their questions may provide valuable insights or signal areas they want to focus on. # Kafka The following notes are all taken from reading [HelloInterview](https://www.hellointerview.com/learn/system-design/deep-dives/kafka): ## Overview When an event happens, the producer creates a message (also called a record) and sends it to a Kafka topic. Each message includes a required **value** field and three optional fields: 1. **Key**: Determines which partition the message goes to. 2. **Timestamp**: Helps to order messages within a partition. 3. **Headers**: Key-value pairs, similar to HTTP headers, used to store metadata about the message. <div align="center"><figure><img src="/files/YPiJCnpu7cbw5TOxUVgw" alt="" width="338"><figcaption>Kafka Message Structure</figcaption></figure></div> **Partition Assignment:** Kafka assigns messages to partitions based on their key. If a message has no key, it uses a round-robin or another set rule. Messages with the same key always go to the same partition, keeping them in order. **Broker Selection:** Kafka identifies which broker handles the partition using cluster data. The producer then sends the message directly to that broker. <figure><img src="/files/dhAZKxcFFOzxo2kR1N2h" alt=""><figcaption>Kafka Architecture</figcaption></figure> ## Terminology ### **Kafka Cluster** * Composed of multiple **brokers** * More brokers = higher scalability for storage and client handling. ### **Broker** * The servers (physical/virtual) that hold the "queue". * Stores data and manages client requests. ### **Partition** * Ordered, immutable sequence of messages, like a log file. * Key for scaling, as partitions enable parallel message consumption. ### **Topic** * Logical grouping of **partitions**. * Used for publishing and subscribing to data. * Supports multiple producers writing data simultaneously. ### **Topic vs Partition** * **Topic**: Logical organization of messages. * **Partition**: Physical organization of messages (can span multiple brokers). ### **Producers and Consumers** * **Producers**: Write data to topics. * **Consumers**: Read data from topics. * Kafka provides APIs for both but leaves message creation/processing to developers. ### **Message Queue vs Stream** * **Message Queue**: Consumers acknowledge messages after processing. * **Stream**: Consumers process messages without acknowledgments, enabling complex processing. # Advanced Topics For scaling Kafka, focus on partitioning (key choice and number of partitions) and adding brokers. For fault tolerance, use replication and track consumer offsets. To improve performance, batch and compress messages, and always think about efficient partitioning. ## **Kafka Broker Constraints**: * A single broker can store **\~1TB** and handle **\~10,000 messages/sec** (depends on hardware). * Keep Kafka messages small **(<1MB)** for optimal performance; Kafka is not for storing large files. * Use Kafka for small messages like pointers (e.g., store large videos in S3, not in Kafka). ## Scalability * **Horizontal Scaling**: Add more brokers to distribute load. Ensure enough partitions to utilize all brokers. * **Partitioning Strategy**: Choose a good key for partitioning (e.g., ad ID). A bad key can cause "hot partitions" (overloaded). * Use random partitioning, salting (adding randomness), or compound keys to handle hot partitions. ## Fault Tolerance & Durability * **Replication**: Each partition is replicated to ensure durability. The replication factor (e.g., 3) defines how many replicas exist. * The replication factor should not exceed the total number of brokers in your cluster. * **Rule of Thumb**: * Ensure the replication factor is less than or equal to the number of brokers. * **Producer Acknowledgments (acks)**: Set `acks=all` for maximum durability—ensures all replicas confirm receipt of a message. * **Consumer Recovery**: Offsets are tracked to ensure consumers can pick up where they left off if they crash. Rebalancing happens automatically if a consumer fails. ## Errors & Retries * **Producer Retries**: Kafka producers automatically retry sending failed messages with a configurable number of attempts. * **Consumer Retries**: Kafka doesn't handle retries for consumers natively, but you can set up a separate "dead letter queue" (DLQ) for retrying or logging failed messages. ## Performance Optimizations * **Batching**: Send messages in batches to reduce overhead. Adjust `maxSize` and `maxTime` for better throughput. * **Compression**: Compress messages (e.g., GZIP) to improve speed by reducing message size. * **Partitioning Strategy**: Ensure even distribution across partitions for better parallelism and throughput. ## Retention Policies * Kafka allows setting a retention period for messages via `retention.ms` (time-based) or `retention.bytes` (size-based). The default is 7 days or 1GB. * If you need longer storage, adjust retention settings—but be mindful of storage costs and performance trade-offs. # URL Shortener When discussing the system design of a URL shortener like TinyURL or Bit.ly, ensure you cover the following key aspects. Structuring your notes around these will help you stay organized during the interview. *** ## Requirements ### **Functional Requirements** * Shorten a given URL and return a unique, shortened version * Redirect users to the original URL when they use the shortened URL. * Custom alias support * Optionally, allow expiration and analytics (clicks, user analytics, etc.). ### **Non-Functional Requirements** * **Focus on availability over consistency**: The service should always be up. * **Low Latency**: Redirection should be fast. * **Scalability**: Support millions or billions of URLs. * **Durability**: Store URLs reliably to prevent data loss. *** ## **Estimation and Capacity Planning** * **QPS (Queries Per Second)**: Estimate reads and writes (e.g., 10:1 ratio). * **Storage Requirements**: Assume average URL length and calculate storage needs (e.g., 1 billion URLs). *** ## Entities * Original URL * Shortened URL * User *** ## **API Design** * **POST `/shorten`** -> returns a shortened URL. * { originalUrl, customAlias?, expirationTime? } * **GET `{shortUrl}`**-> Redirects to the original URL. * **HTTP 302 (temporary redirect)** instead of HTTP 301 (permanent redirect which browser caches) * Optional APIs for stats or management: * **GET `/stats/{shortUrl}`**. **b) Database Schema** * `ShortUrl (short_key, original_url, created_at, expiration_date)` * Index on `short_key` for fast lookups. **c) URL Shortening Logic** * **Short Key Generation**: * Use a **Base62** encoding scheme (characters `a-z`, `A-Z`, `0-9`) to keep keys short and human-readable. * A 6-character Base62 key can represent over 56 billion unique URLs (62⁶). * **Counter/Sequence**: Increment a global counter and encode the value in Base62. * A single Redis instance on modern hardware can handle **80,000 to 100,000 operations per second (OPS)** for simple commands like `INCR`, assuming: * Redis is running on dedicated high-performance hardware. * The network is low-latency and high-bandwidth. * There are no significant memory or disk I/O bottlenecks. * Handle **collisions** in case of duplicate keys. **d) Data Storage** * Choose a storage system based on scale: * **Relational Databases**: MySQL/PostgreSQL for small-scale systems. * **NoSQL Databases**: DynamoDB, Cassandra, or MongoDB for large-scale systems. * In-memory caching (Redis or Memcached) for frequently accessed data. **e) Redirection** * Use **HTTP 301 (Moved Permanently)** or **302 (Found)** for redirection. *** ## **Scaling the System** **a) Read/Write Patterns** * Reads will dominate writes (many more redirections than shortenings). * Optimize for high-read throughput using caching. **b) Partitioning and Replication** * Use database replication for durability, fault tolerance and to allow scaling reads horizontally. * Partition data by the short key to distribute load (consistent hashing). **c) Caching** * Use Redis or Memcached to cache mappings of frequently accessed `short_key -> original_url`. **d) Rate Limiting** * Implement rate limiting to prevent abuse (e.g., spamming the shorten API). *** ## **High-Level Architecture** <figure><img src="/files/b1tpY7uEIx7dfdRHz1Ou" alt=""><figcaption></figcaption></figure> *** ## **Trade-offs and Challenges** * **Key Length**: Shorter keys are user-friendly but increase the chance of collisions. * **Consistency**: Balancing consistency and availability (CAP theorem). * **Redundancy**: Ensure data is not lost due to server failures. * **Performance**: Optimize key lookup and redirection latency.

# Introduction These pages cover most of the notes I consider useful and that I've taken over my years learning to become a professional in the field of Software Engineering. Visit to go to my homepage. # System Preferences MacOS preferences ### Trackpad * *Point & Click* * Enable *Tap to click with one finger* * *Scroll & Zoom* * Disable S*croll direction natural* * *More Gestures* * *Swipe between full-screen apps* * *Swipe left or right with four fingers* ### Accessibility * *Zoom* * Enable *Use scroll gesture with modifier keys to zoom* * *Pointer Control* * *Trackpad Options* * *Enable Dragging* * *Three finger drag* ### Sound * Enable *Show volume in menu bar* ### Network * Enable *Show Wi-Fi status in menu bar* ### Bluetooth * Enable *Show Bluetooth in menu bar* # Homebrew The Missing Package Manager for macOS (or Linux). ### MacOS Requirements * A 64-bit Intel CPU or Apple Silicon CPU [1](https://docs.brew.sh/Installation#1) * macOS Mojave (10.14) (or higher) [2](https://docs.brew.sh/Installation#2) * Command Line Tools (CLT) for Xcode: ``` xcode-select --install ``` * A Bourne-compatible shell for installation (e.g. `bash` or `zsh`) [4](https://docs.brew.sh/Installation#4) ## Installation To install Homebrew run the following in a terminal: ``` /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" ``` hit **Enter**, and follow the steps on the screen. # Usage To install a package (or **Formula** in Homebrew vocabulary) simply type: ``` brew install ``` To update Homebrew's directory of formulae, run: ``` brew update ``` To see if any of your formulas need to be updated: ``` brew outdated ``` To update a formula: ``` brew upgrade ``` Homebrew keeps older versions of formulas installed on your system, in case you want to roll back to an older version. That is rarely necessary, so you can do some cleanup to get rid of those old versions: ``` brew cleanup ``` If you want to see what formulae Homebrew would delete *without actually deleting them*, you can run: ``` brew cleanup --dry-run ``` To see what you have installed (with their version numbers): ``` brew list --versions ``` To search for formulas you run: ``` brew search ``` To get more information about a formula you run: ``` brew info ``` To uninstall a formula you can run: ``` brew uninstall ``` # iTerm iTerm is an open source replacement for Apple's Terminal. It's highly customizable and comes with a lot of useful features. ### Installation Use [Homebrew](https://sourabhbajaj.com/mac-setup/Homebrew/) to download and install: ``` brew install --cask iterm2 ``` # VIM Vim is a highly configurable text editor built to make creating and changing any kind of text very efficient. It is included as "vi" with most UNIX systems and with Apple macOS. ### Installation To install the latest version, use homebrew: ``` brew install vim ``` ### The Ultimate vimrc [The Ultimate vimrc](https://github.com/amix/vimrc) it's a collection of vimrc configurations to make easy the usage of vim. To download the The Ultimate vimrc, you need to install the git client. If you need install it, use home brew: ``` brew install git ``` Now, download the vimrc files: ``` git clone https://github.com/amix/vimrc.git ~/.vim_runtime ``` To install the complete version, run: ``` sh ~/.vim_runtime/install_awesome_vimrc.sh ``` To install the *basic* version, run: ``` sh ~/.vim_runtime/install_basic_vimrc.sh ``` #### Update To update the vimrc scripts, run: ``` cd ~/.vim_runtime && git pull --rebase && cd - ``` ### Maximum Awesome [Maximum Awesome](https://github.com/square/maximum-awesome) it's a collection of vim configuration and plugins, like a configuration manager for the vim environment. #### Installation To install it, just make a clone of the repository with the git client: ``` git clone https://github.com/square/maximum-awesome.git ``` Then install it: ``` cd maximum-awesome rake ``` > **NOTE:** the rake command will install all dependencies needed. # Tree ### Installation To install the latest version, use homebrew: ``` brew install tree ``` ### Usage Running `tree` will produce output like this: ``` $ tree . ├── Apps │ ├── Octave.md │ ├── README.md │ ├── Settings.md │ ├── araxis-merge.jpg │ ├── beyond-compare.png │ ├── delta-walker.jpg │ ├── filemerge.png │ └── kaleidoscope.png ├── CONTRIBUTING.md ├── Cpp │ └── README.md ├── Docker │ └── README.md ├── Git │ ├── README.md │ └── gitignore.md └── Go └── README.md 5 directories, 14 files ``` To limit the recursion you can pass an `-L` flag and specify the maximum depth `tree` will use when searching. ``` tree -L 1 ``` will output: ``` . ├── Apps ├── CONTRIBUTING.md ├── Cpp ├── Docker ├── Git └── Go 5 directories, 1 files ``` # ZSH The Z shell (also known as `zsh`) is a Unix shell that is built on top of `bash` (the default shell for macOS) with additional features. It's recommended to use `zsh` over `bash`. It's also highly recommended to install a framework with `zsh` as it makes dealing with configuration, plugins and themes a lot nicer. We've also included an `env.sh` file where we store our aliases, exports, path changes etc. We put this in a separate file to not pollute our main configuration file too much. This file is found in the bottom of this page. Install `zsh` using Homebrew: ``` brew install zsh ``` Now you should install a framework, we recommend to use [Oh My Zsh](https://github.com/robbyrussell/oh-my-zsh) or [Prezto](https://github.com/sorin-ionescu/prezto). **Note that you should pick one of them, not use both.** The configuration file for `zsh` is called `.zshrc` and lives in your home folder (`~/.zshrc`). ### Oh My Zsh [Oh My Zsh](https://github.com/robbyrussell/oh-my-zsh) is an open source, community-driven framework for managing your `zsh` configuration. It comes with a bunch of features out of the box and improves your terminal experience. Install Oh My Zsh: ``` sh -c "$(curl -fsSL https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh)" ``` The installation script should set `zsh` to your default shell, but if it doesn't you can do it manually: ``` chsh -s $(which zsh) ``` #### Configuration The out-of-the-box configuration is usable but you probably want to customise it to suit your needs. The [Official Wiki](https://github.com/robbyrussell/oh-my-zsh/wiki) contains a lot of useful information if you want to deep dive into what you can do with Oh My Zsh, but we'll cover the basics here. To apply the changes you make you need to either **start new shell instance** or run: ``` source ~/.zshrc ``` **Plugins** Add plugins to your shell by adding the name of the plugin to the `plugin` array in your `.zshrc`. ``` plugins=(git colored-man-pages colorize pip python brew osx zsh-syntax-highlighting zsh-autosuggestions) ``` You'll find a list of all plugins on the [Oh My Zsh Wiki](https://github.com/robbyrussell/oh-my-zsh/wiki/Plugins). Note that adding plugins can cause your shell startup time to increase. # Visual Studio Code Visual Studio Code is a lightweight code editor with support for many programming languages through extensions ## Installation To install the latest version, use Homebrew: ``` brew install --cask visual-studio-code ``` ### macOS integration Launch VS Code from the [command line](https://code.visualstudio.com/docs/setup/mac#_launching-from-the-command-line). After that, you can launch VS Code from your terminal: * `code .` will open VS Code in the current directory * `code myfile.txt` will open `myfile.txt` in VS Code ### Useful Extensions #### Python * [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python) - Python code highlighting To enable auto-formatting on "Save", i.e. `⌘ + S`, configure the following: 1. Change the default formatter to `Black` instead of `Autopep8`. Critical to avoid large diffs. Go to *Preferences* -> *User Settings* and update the setting `python.formatter.provider` to `Black` 2. Enable `Format on Save` Setting: *Editor: Format On Save* setting on *Code* -> *Preferences* -> *Settings* #### JavaScript * [ESLint](https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint) - Useful to check JavaScript errors and helps in auto-formatting the code * [Prettier](https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode) - JavaScript code formatter #### SQL * [PostgreSQL formatter](https://marketplace.visualstudio.com/items?itemName=bradymholt.pgformatter) #### Markdown * [Markdown Preview](https://marketplace.visualstudio.com/items?itemName=shd101wyy.markdown-preview-enhanced) - Read Markdown files in VSCode #### GitLens * [GitLens](https://marketplace.visualstudio.com/items?itemName=eamodio.gitlens) - Supercharge the Git capabilities built into VSCode #### Docker * [Docker](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker) - Create, manage, and debug images from within VSCode #### JSON * [Paste JSON as Code](https://marketplace.visualstudio.com/items?itemName=quicktype.quicktype) - Infers types from sample JSON data, then outputs strongly typed models and serializers for working with that data in your desired programming language * [Live Server](https://marketplace.visualstudio.com/items?itemName=ritwickdey.LiveServer) - Launches a local development server with live reloading for both static and dynamic #### VS Code Icons * [vscode-icons](https://marketplace.visualstudio.com/items?itemName=vscode-icons-team.vscode-icons) - Adds unique icons to distinguish different file extensions (easier to glance through your directories) # Git Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. Install using Brew: ``` brew install git ``` When done, to test that it installed properly you can run: ``` git --version ``` And `which git` should output `/usr/local/bin/git`. Next, you can define your global Git user: ``` git config --global user.name "Your Name Here" git config --global user.email "your_email@youremail.com" ``` They will get added to your `.gitconfig` file found in the home folder `~/`. # SSH Keys ## What are SSH Keys SSH (Secure Shell) keys are cryptographic keys used to securely connect to a server without needing to enter a password. They consist of a pair of keys: a **public key** and a **private key**. ### Algorithms Different algorithms can be used to generate SSH keys. Each has its own strengths and weaknesses, depending on the level of security and compatibility needed. #### **Choosing the Right Algorithm** * **RSA:** Choose if you need broad compatibility, especially with older systems. * **ED25519:** Recommended for most users due to its security and efficiency. * **ECDSA:** Consider if you need high security with smaller key sizes for specialized environments. ## How to Generate an SSH Key 1. **Open a terminal.** 2. **Generate the key:** * For ED25519 (recommended): ```mathematica ssh-keygen -t ed25519 -C "your_email@example.com" ``` 3. **Follow the prompts:** * When asked where to save the key, press Enter to accept the default location unless you need to save it elsewhere. * Set a passphrase if you want extra security. 4. **Final Steps:** * Your public and private keys are now generated. * You can add it to your Git platforms (GitHub, Gitlab, etc.) \
# How To Measure ### The Balanced Approach: Qualitative + Quantitative Developer productivity isn't just about lines of code or tickets closed. It's a complex interplay of efficiency, quality, and satisfaction. The most effective measurement approach combines **quantitative metrics** (the numbers) with **qualitative data** (the human experience). Why this combination works best: * Quantitative data provides concrete benchmarks and trends * Qualitative insights reveal the "why" behind the numbers * Together, they create a holistic view of developer experience and output ### Evolution of Developer Productivity Metrics #### DORA Metrics (2018)

https://cloud.google.com/blog/products/devops-sre/another-way-to-gauge-your-devops-performance-according-to-dora

The DevOps Research and Assessment (DORA) team at Google, led by Dr. Nicole Forsgren, Jez Humble, and Gene Kim, introduced four key metrics that have become industry standards: * **Deployment Frequency**: How often code is deployed to production * **Lead Time for Changes**: Time from code commit to production deployment * **Mean Time to Recovery (MTTR)**: How quickly service is restored after failures * **Change Failure Rate**: Percentage of deployments causing failures DORA metrics focus on delivery performance and have been validated through rigorous research across thousands of teams. They're primarily quantitative but reveal important patterns in team effectiveness. #### SPACE Framework (2021)

Developed by researchers from GitHub, Microsoft, and the University of Victoria (including Nicole Forsgren, Margaret-Anne Storey, and Eirini Kalliamvakou), SPACE offers an approach with five dimensions: * **Satisfaction and well-being**: Developer happiness and health * **Performance**: Quality and impact of work * **Activity**: Design, coding, review, and collaboration actions * **Communication and collaboration**: How teams work together * **Efficiency and flow**: Ability to work without blockers SPACE deliberately balances quantitative measures with qualitative experiences, recognizing that productivity is multidimensional. #### DevEx (2023)

https://www.infoq.com/articles/devex-metrics-framework/

The Developer Experience (DevEx) framework emerged from research by McKinsey and the SPACE team to focus specifically on the developer experience: * **Feedback loops**: How quickly developers can validate their work * **Cognitive load**: Mental effort required for daily tasks * **Flow state**: Ability to work without interruptions DevEx represents the latest evolution in productivity thinking, acknowledging that developer experience directly impacts output quality and velocity. ### Making Measurement Work The most effective approach is deliberately combining methods: 1. **Start with clear goals**: What specifically are you trying to improve? 2. **Mix data types**: Pair metrics (DORA) with experience surveys (SPACE/DevEx) 3. **Contextualize the numbers**: A metric without context is just a number 4. **Measure consistently**: Track changes over time, not just snapshots 5. **Involve the team**: Developers should help define what productivity means ### Setting Effective Targets Setting targets for developer productivity metrics requires careful consideration to avoid distorting the system you're trying to improve. Research shows that "when a measure becomes a target, it ceases to be a good measure" (Goodhart's Law). #### Focus on Input Metrics, Not Outputs The key to effective target-setting is distinguishing between: * **Output metrics**: Results you want to improve but can't directly control (PR throughput, change failure rate) * **Controllable input metrics**: Specific behaviors and processes teams can directly influence (code review turnaround time, build time) Never set targets on output metrics. This creates confusion and encourages gaming the system. Instead, identify the controllable inputs that drive those outputs and focus your targets there. #### Preventing Metric Gamification To prevent teams from "gaming" metrics: * Use multi-dimensional measurement (never rely on a single metric) * Emphasize learning and improvement over hitting specific thresholds * Give teams sufficient time to make systemic improvements * Involve developers in selecting metrics and setting realistic targets * Make the connection between input metrics and business outcomes clear #### Best Practices for Target Setting 1. **Limit focus**: Target 5-6 controllable input metrics at most 2. **Set contextual targets**: Different teams need different targets based on their unique challenges 3. **Be realistic about improvement curves**: Moving from the 75th to 90th percentile is much harder than from the 50th to 75th 4. **Consider metric types**: Some metrics need percentage improvements, others need absolute targets or SLAs 5. **Create transparency**: Clearly explain how metrics will be used and decisions they'll inform ### Making Measurement Work The most effective approach is deliberately combining methods: 1. **Start with clear goals**: What specifically are you trying to improve? 2. **Mix data types**: Pair metrics (DORA) with experience surveys (SPACE/DevEx) 3. **Contextualize the numbers**: A metric without context is just a number 4. **Measure consistently**: Track changes over time, not just snapshots 5. **Involve the team**: Developers should help define what productivity means ### Bottom Line The best productivity measurement combines quantitative metrics with qualitative data. Neither approach alone tells the complete story. Together, they provide actionable insights that drive meaningful improvements in how developers work. Your measurement approach should be as thoughtful and multifaceted as the work you're measuring. # SRE This page and its sub-pages contain my notes from studying the Google SRE workbooks: **How Google Runs Production Systems:** **Practical Ways to Implement SRE:** \
## **Pre-DevOps** **Operations Challenges**: Complex, context-dependent; often treated as cost center in enterprises. ## **DevOps** * **Principles**: CALMS (Culture, Automation, Lean, Measurement, Sharing) * **Focus**: Collaboration, continuous improvement, no silos. * **Key Ideas**: * Accidents are normal and expected. * Gradual, small changes preferred. * Culture over tooling for success. * Measurement crucial for improvement. ## **SRE** * **Definition**: Implementing DevOps philosophy with a focus on concrete practices. * **Principles**: * Operations is a software problem. * Manage by Service Level Objectives (SLOs). * Minimize toil; automate where possible. * Wisdom of production informs design. * Reduce cost of failure to enhance development speed. * Share ownership with developers. * Unified tooling across roles. ## **Comparison** * **Similarities**: * Acceptance of change. * Collaboration and shared ownership. * Small, continuous changes. * Importance of measurement and blameless postmortems. * Holistic approach to improvement. * **Differences**: * DevOps: Broader, culture-focused; not detailed in service management. * SRE: Service-specific, structured around detailed principles like SLOs and error budgets. # Scaling Reliably How do we know if a system is scaling reliably? A system is considered reliable if it can consistently perform and meet requirements despite changes in its environment over time. Achieving this requires the system to detect failures, self-heal automatically, and scale according to demand. The CAP theorem is a conclusion made by computer scientist Eric Brewer. It's a popular and fairly useful way to think about tradeoffs in the guarantees that a system design makes.

* Consistency: all nodes see the same data at the same time. * Availability: node failures do not prevent survivors from continuing to operate. * Partition tolerance: the system continues to operate despite message loss due to network and/or node failure

## Configuration Management ### **Terraform**: * Use to define infrastructure as code (IaC). * **Terragrunt** to keep things DRY (Don't repeat yourself).

* Example: [Multi-region deployments with provider configurations](https://github.com/chrismarget/multi-region-terraform-example) (using **alias**). ``` # The default provider configuration; resources that begin with `aws_` will use # it as the default, and it can be referenced as `aws`. provider "aws" { region = "us-east-1" } # Additional provider configuration for west coast region; resources can # reference this as `aws.west`. provider "aws" { alias = "west" region = "us-west-2" } ``` ### **Ansible/Chef/Puppet**: * Automate configuration and ensure consistency across environments. ### **Git**: * Version control for infrastructure and configuration files. * Example: Gitops CI/CD (merge to main deploys to one environment, git tags deploy to another, etc.) ### **LaunchDarkly:** * **User Segmentation:** Target specific users or user segments based on attributes such as location, role, or subscription tier. * **Custom Conditions:** Use logic to roll out the feature only to users who meet specific criteria. * **Percentage Rollouts:** Gradually enable the feature for a small percentage of users and incrementally increase the percentage. *** ## Kubernetes (K8s) ### **Multi-region Clusters**: * Use cluster federation for workload distribution (**KubeFed is the official tool**).

* **What:** Cluster federation is a method in Kubernetes to combine multiple clusters across different regions into a single super-cluster that can be controlled through a single interface. * The Kubernetes Control Plane handles apps across a group of worker nodes. The Federated Control Plane does the same thing, but it does it across a group of clusters instead of nodes. * **Why:** Allows you to schedule workloads (apps/pods) dynamically across clusters based on factors like geographic location, resource availability, or latency requirements. * Aims to provide **high availability** * **Examples**: If you have users in North America and Europe, federation can ensure workloads are distributed to clusters closest to users, reducing latency. * **Namespace**: Creates a namespace with the same name in all clusters. * **ConfigMap**: Replicated in all clusters within the same namespace. * **Deployments/ReplicaSets**: Adjusted to distribute replicas fairly across clusters (can be customized). * A federated Deployment with 10 replicas spreads these 10 pods fairly across clusters (not 10 pods per cluster). * **Ingress**: Creates a unified access point across clusters, not individual Ingress objects. * Set up regional failover with DNS routing (e.g., Route 53). ### **Scheduling Pods Across Nodes and AZs** * This method helps distribute replicas across multiple nodes and availability zones (AZs) for better fault tolerance and availability. * **Anti-Affinity**: Add `podAntiAffinity` to a **Deployment** resource to avoid placing replicas on the same node or availability zone. * **Weight**: `weight: 100` increases the preference for spreading across AZs. #### Kafka-Specific Deployments * **Partitioning**: * Distribute partitions across brokers for load balancing. * **Replication**: * Configure replication factor for fault tolerance. * Multi-region replication for disaster recovery. * **Producers/Consumers**: * Optimize producer ACK settings for latency vs. durability. * Monitor consumer lag using Kafka Connect. *** ## Observability ### Logging * **Centralized Systems**: * ELK Stack (Elasticsearch, Logstash, Kibana). * Splunk for large-scale log management. * **Structured Logging**: * Use JSON formatting for better parsing. * Include trace IDs for distributed tracing. * **Retention Policies**: * Configure log rotation and archive policies to manage costs. ### Monitoring #### **What to monitor** * Four Golden Signals: Latency, Traffic, Error rates, Saturation. * **Errors** golden signal measures the rate of requests that fail. * Usually monitoring systems focus on the error rate, calculated as the percent of calls that are failing from the total. * **Latency** is defined as the time it takes to serve a request. * Use Histograms (Ex: P99) * Talk by Gil Tene: * "Typical user sessions involve 5 page loads with over 40 resources per page" * Given this, some percentiles like P95 might be useless to monitor * Instead of focusing solely on a few percentiles (e.g., P50, P90, P99), use **HDR histograms** (High Dynamic Range histograms) to capture the full distribution of latencies. These histograms can show the range of latencies across the system. * Visualize **saturation** and understand the resources involved in your solution. What are the consequences if a resource gets depleted? * **Traffic** measures the amount of use of your service per time unit. * The amount of users by time of day might be relevant for example * **Healthchecks**: * Use Startup/Readiness/Liveness/ probes * Startup: For containers that take a long time to start * Readiness: db down, mandatory auth down, app can only serve X max connection * In many cases, a `readinessProbe` is all we need * Liveness: Deadlocks, internal corruption, etc. * *This container is dead, we don't know how to fix it* * Types of probes: * httpGet, exec, tcpSocket, grpc, etc. * All probes return a binary result (success/fail) * You can add a “ping” endpoint for an HTTP service that returns a 200 when called. * **Infrastructure Metrics** (CPU, Memory, IO, Network) * **Business Metrics**: Things like the number of completed orders #### **Tools** * **Prometheus** is an open-source monitoring system including: * multiple *service discovery* backends to figure out which metrics to collect * a *scraper* to collect these metrics * an efficient *time series database* to store these metrics * a specific query language (PromQL) to query these time series * an *alert manager* to notify us according to metrics values or trends * Grafana for visualization. * Datadog/CloudWatch for cloud-native monitoring. **Key Metrics**: * SLIs: Latency, Traffic, Error rates, Saturation. * SLOs: Define acceptable thresholds (e.g., 99.9% uptime). * Identify what users care about most when interacting with the service. Common factors include: * **Availability**: Users expect services to be available when needed. * **Latency**: Users want fast responses. * **Correctness**: Users expect the service to function as intended without errors.

* SLAs: Contracts outlining the consequences if SLOs aren’t met * **Distributed Tracing**: * Tools: Jaeger, OpenTelemetry. * Trace requests across microservices to identify bottlenecks. ### Alerting * **Strategies**: * Use anomaly detection for proactive alerts. * Tiered alerts to reduce noise (e.g., warnings vs. critical). * **Tools**: * PagerDuty, Opsgenie for incident response. *** ## **Load Balancers** Load Balancers are essential for horizontally scaling applications. They distribute incoming traffic across multiple backend resources.

Difference between Load Balancer and Reverse Proxy

#### **Advantages of Load Balancing:** * Load balancers minimize server response time and maximize throughput. * Load balancer ensures high availability and reliability by sending requests only to online servers * Load balancers do continuous health checks to monitor the server’s capability of handling the request. **Types of Load Balancer Algorithms:**

**Types of Load Balancers:** [**https://www.geeksforgeeks.org/layer-4-load-balancing-vs-layer-7-load-balancing/?ref=next\_article**](https://www.geeksforgeeks.org/layer-4-load-balancing-vs-layer-7-load-balancing/?ref=next_article) 1. **Network Load Balancer (NLB):** * **Layer**: Layer 4 (Transport) * **Function**: Forwards incoming traffic to backend targets (e.g., EC2 instances, pods) without modifying the packet headers, preserving the source IP. * **Use Case**: Ideal for **TCP/UDP traffic**. 2. **Application Load Balancer (ALB):** * **Layer**: Layer 7 (Application) * **Function**: Can inspect HTTP/HTTPS requests, enabling routing based on URL paths, headers, and other attributes. Essentially, it acts as a reverse proxy. (HAProxy, NGINX) * **Use Case**: Best for **HTTP traffic**, with support for advanced routing rules. *** #### **How This Applies to EKS (Kubernetes)** In an EKS environment, AWS Load Balancers work with Kubernetes Services and Ingress objects to manage traffic routing. * **Service of type LoadBalancer**: When you create a Service with this type in Kubernetes, the **AWS Load Balancer Controller** sets up an **NLB** to manage Layer 4 traffic. * **Ingress Object**: If you create an Ingress object, the controller sets up an **ALB** to manage Layer 7 traffic, supporting features like path-based routing or hostname-based routing to specific services. *** #### **Multi-Region Load Balancing** 1. **High Availability**: Multi-region deployments ensure that your application can continue operating even if one region faces issues (e.g., outages, natural disasters). 2. **Latency Optimization**: Routing traffic to the closest region reduces latency, improving performance. 3. **Route 53 Latency-Based Routing**: AWS Route 53 can route traffic to the region with the lowest latency for users, enhancing the user experience. *** #### **Reliability and Failover Mechanisms** 1. **Load Balancer Performance**: Access logs provide insights into traffic distribution, latency, and errors. This data is crucial for monitoring performance and troubleshooting. 2. **Security and Compliance**: Logs help in detecting security threats (e.g., DDoS attacks) and ensuring compliance with standards like PCI-DSS or HIPAA. 3. **Health Checks**: Both ALB and NLB use health checks to verify if a target (pod, EC2 instance) is healthy. Unhealthy targets are automatically excluded from traffic routing, ensuring reliability. 4. **Auto Scaling**: Kubernetes and AWS auto scaling mechanisms add more instances or pods when traffic increases, maintaining application performance. 5. **Cross-AZ Failover**: Multi-AZ deployments automatically reroute traffic to healthy resources in other Availability Zones if one AZ becomes unavailable, ensuring high availability. ## Databases ### **Performance Optimization** * **Slow Queries** * Use query analyzers to identify slow queries. * Add custom tags or identifiers in SQL queries for tracking application-level operations. * Use indexes, avoid full table scans, and rewrite expensive queries. * **Key Metrics to Monitor** * **CPU**: Track overall CPU usage and per-query CPU utilization for performance insights. * **IOPS**: Monitor read/write to evaluate workload distribution and storage performance. * **Optimize Data Access**: Reduce unnecessary reads/writes by denormalizing data, partitioning tables, or archiving unused data. * **Increase IOPS Capacity**: Use faster storage (e.g., SSDs over HDDs) or provision higher IOPS for better performance. * **Buffering/Batching**: Group writes into larger, less frequent operations to reduce I/O operations * **Scaling**: * **Vertical Scaling**: Increase instance size for better resource allocation. * **Horizontal Scaling**: Add replicas or shards for improved database distribution. * **Caching**: Use Redis or Memcached to offload repeated queries and reduce load on the database. * **Change Data Capture (CDC):** CDC enables efficient synchronization of data between systems without requiring heavy full-database reads or disruptive batch updates. * Example of using CDC for a search service that uses ElasticSearch

### **Reliability and Availability Enhancements** * **Database Replication (Master-Slave Structure)** * Pros: Improves performance, reliability and availability] * Cons: Small delay to replicate data in slave DBs

* **Master Database**: Handles all write operations and propagates changes to slaves. * **Slave Databases**: Handle read operations, helping distribute the load and improve reliability. *** ## Common Interview Scenarios and Responses 1. **Multi-Region Deployment**: * Use DNS-based routing (e.g., Route 53) for global traffic management. * Deploy services in active-active configuration across regions. * Synchronize data with replication (e.g., Kafka, database replicas). 2. **Database Scaling**: * Vertical scaling: Add more resources (CPU/RAM) temporarily. * Horizontal scaling: Add read replicas for increased throughput. * Monitor replica lag and implement query routing strategies. 3. **Logging and Monitoring**: * Centralize logs using ELK. * Set up Prometheus alerts for latency spikes. * Use Jaeger to trace high-latency requests. ## Troubleshooting Bottlenecks ### **Scenario**: High Latency in API Requests * Check logs for errors or slow endpoints. * Use APM to profile services. * Add caching or optimize database queries. ### **Scenario**: Replica Lag in Database * Analyze replication metrics. * Offload read-heavy queries to replicas. * Increase IOPS capacity if disk throughput is limited. ### **Scenario**: Traffic Spike in a Region * Autoscale services. * Redirect traffic using load balancers. * Enable rate limiting to prevent overload. # Splitting a Monolith into Microservices **Introduction** * **Core Idea**: Moving from a monolithic architecture to microservices involves breaking down applications into smaller, independently deployable units. * **Challenge**: Transitioning to a distributed system introduces complexities like inter-service communication, data sharing, and failure modes. * **Goal**: Enable scalability—not just technically but organizationally—by allowing teams to work autonomously. *** ## **Why Split a Monolith?** 1. **Scalability in Team Dynamics**: * Microservices allow teams to iterate independently. * Each service has its own deployment cycle, reducing bottlenecks in large teams. 2. **Addressing Real-World Distribution**: * Businesses are inherently distributed systems; services model this reality. * Enables better integration between interdependent systems (e.g., Single Sign-On, Order Management). 3. **Avoid Coordination Overhead**: * Monolithic changes are easier to synchronize but lead to rigidity. * Microservices allow loosely coupled, bounded contexts that can evolve independently. *** ## **Challenges in Splitting Monoliths** ### Ecommerce Example #### **Orchestrator** Orchestration-based sagas are easier to understand and manage for complex processes. However, they introduce a central point of control, which can be a bottleneck or single point of failure if not designed carefully. And the communication between the Orchestrator and the rest of the services is still complex and susceptible to errors. * **GCP Workflows** is ideal for **lightweight, event-driven, API-first automation** where you need to quickly integrate services or microservices (e.g., processing user requests, serverless automation). 1. The Orchestrator (GCP Workflows) receives the order request. * Workflow calls the Order Service API to create an order. * Calls the Inventory Service API to reserve items. * Calls the Payment Service API to charge the customer. * Sends a notification through an email/SMS API.

**Cons**: * Can create a single point of failure if the orchestrator goes down. * Less flexible and scalable since the orchestrator must handle all communication. * Adds overhead in terms of managing and scaling the central controller. #### Choreography In Choreographed Transactions there is no central coordinator. Instead, each service listens for events from other services and decides when to execute its local transaction, or the necessary corrective actions. More importantly, each service decides what are the correct actions to execute. For example, in an e-commerce system: 1. The Order service receives a message, creates an order, and publishes an "OrderCreated" event. 2. The Inventory service listens for "OrderCreated", reserves the items, and publishes "InventoryReserved". 3. The Payment service listens for "InventoryReserved", processes the payment, and publishes "PaymentProcessed". 4. The Order service listens for "PaymentProcessed" and marks the order as complete. If any step fails (e.g., payment fails), the service publishes a failure event. Other services listen for this and execute compensating actions (e.g., Inventory releases the reserved items).

To implement a choreographed transaction you would typically use services like **Kafka**. While something like **GCP Pub/Sub** could be used, it operates in an **eventual consistency** model, which may not be suitable for some use cases requiring **strong consistency** across services. **Pros**: * Decentralized, with no single point of failure. * More scalable and flexible, as services interact directly with each other. * Reduces bottlenecks by eliminating the need for a central controller. #### Idempotency There is no "exactly once" delivery of events. There is either "at most once" or "at least once". To ensure no messages are lost, you'll want "at least once". That means your services need to be prepared to receive duplicates, and either be able to identify and drop them, or only execute idempotent actions. By the way, idempotency means that an action can be executed an arbitrary number of times and it will have the same result as if it was executed once. Ideally all actions in all systems should be idempotent, but in event-driven systems it's even more important. Fortunately, event IDs help a lot with this. ### **The Data Dichotomy**: * **Conflict**: * Services encourage encapsulation (hiding data). * They expose only a limited set of functionalities through APIs or interfaces, while hiding its internal data and implementation details. * Databases and storage systems promote making data accessible. * Usually centralized and easy for multiple applications to access. * **Impact**: Data across services may become inconsistent and harder to sync/align. 1. **Shared Data Complexities**: * Inter-service dependencies on common datasets lead to **tight coupling**. * ```

``` * Duplicated or locally altered data can diverge, causing consistency issues. 2. **Design Flaws to Avoid**: * **"Kookie" Databases**: Service interfaces that evolve into complex, shared databases. * (A) Service interfaces are poorly suited to sharing data at any level of scale * (B) Messaging moves data, but provides no historical reference, and this leads to data corruption over time. * (C) Shared databases concentrate too much in one place

*** ## **Solution: Event-Driven Architectures and Streaming Platforms** 1. **Distributed Log** (e.g., Apache Kafka): * Acts as a central source for immutable event streams. * Retains historical data for reproducibility and recovery. 2. **Stateful Stream Processing**: * Services process shared datasets locally while keeping the "golden source" in the log. * Combines the benefits of encapsulation with data accessibility. 3. **Advantages**: * **Balance**: Resolves the data dichotomy by keeping the data immutable and operations decentralized. * **Flexibility**: Services can join and process streams as needed without polluting the source. * **Consistency**: Reduces data divergence by maintaining a regenerable source of truth. *** ## **Practical Steps to Splitting Monoliths** 1. **Identify Boundaries**: * Break down domains into bounded contexts (e.g., SSO, Order Processing). 2. **Define Interfaces**: * Use APIs or messaging systems for inter-service communication. 3. **Adopt Event-Driven Patterns**: * Replace synchronous dependencies with event streams to decouple services. 4. **Incremental Transition**: * Gradually replace monolithic components with microservices. * Start with non-critical features to avoid disruption. 5. **Centralize Streams, Decentralize Logic**: * Use a distributed log for event storage and stream processing engines for business logic. *** ## **Key Tools and Concepts** * **Apache Kafka**: * Stores event streams for long-term, immutable data sharing. * Facilitates integration with connectors to export/import data on demand. * **Stateful Stream Processing**: * Embeds database-like processing within services to maintain independence. * **Decentralized Processing**: * Encapsulate function (not data) within services, ensuring autonomy. *** ## **Summary for Interview** When asked about splitting a monolith: * **Begin**: Highlight why companies adopt microservices: scalability, adaptability, and better alignment with distributed business realities. * **Explain Challenges**: Discuss the data dichotomy, shared data issues, and the pitfalls of shared databases or batch transfers. * In the real world, business services won't be able to have an easy separation of concerns.

* **Present Solutions**: Advocate for event-driven architectures with distributed logs like Kafka, emphasizing its role in balancing encapsulation and accessibility. * **Provide a Roadmap**: Outline an incremental approach—starting with identifying domains, defining interfaces, and adopting event-driven patterns. # Troubleshooting Common Issues

## **Cannot Reach Server** 1. **Ping the server by Hostname and IP Address:** * **Hostname/IP Address is pingable:** * The issue might be on the client side since the server is reachable. * **Hostname is not pingable but IP Address is pingable:** * Likely a DNS issue. Check: * `/etc/hosts` * `/etc/resolv.conf` * `/etc/nsswitch.conf` * **Test DNS Resolution:** * **Using `nslookup, dig or host`** * **Neither Hostname nor IP Address is pingable:** * Check another server on the same network: * **False:** Issue is with this specific host/server. * **True:** Likely a broader network issue. * Log in via Virtual Console (if the server is powered on): * Check uptime using command `uptime`. * Verify if the server has an IP and if the network interface is UP. * Run the command `ip addr` * Ensure the network interface (e.g., `eth0`, `ens33`) is listed and in the "UP" state. * Ping the gateway and check routes. * Check SELinux and firewall rules. * Inspect physical cable connections. ## **Cannot Reach Website or Application** 1. **Ping the server by Hostname and IP Address:** * **False:** Follow troubleshooting steps from “Server is not reachable or unable to connect.” * **True:** Check service availability using the `telnet` command with the appropriate port: * **True:** The service is running. * **False:** The service is not reachable or running. Check: * Service status (using `systemctl` or equivalent commands). * Firewall/SELinux settings. * Service logs. * Service configuration. ## **Unable to SSH as Root or User** 1. **Ping the server by Hostname and IP Address:** * **False:** Follow troubleshooting steps from “Cannot Reach Server” * **True:** Check service availability using the `telnet` command with the SSH port: * **True:** The service is running: * Check if the issue is on the client side. * Verify: * User account is not disabled. * User has a valid shell (not `nologin`). * Root login is not disabled in the SSH configuration. * **False:** The service is not reachable or running. Check: * Service status (using `systemctl` or equivalent commands). * Firewall/SELinux settings. * Service logs. * Service configuration. ## **Disk Space is Full or Adding/Extending Disk Space** 1. **Detect Performance Degradation:** * Applications are slow or unresponsive. * Commands fail to execute (e.g., `/` disk space is full). * Logging and other system operations fail. 2. **Analyze the Issue:** * Use the `df` command to identify the problematic filesystem. 3. **Take Action:** * Use `du` to find large files/directories in the affected filesystem. * Compress or remove large files. * Move files to another partition or server. * Check disk health with `badblocks` (e.g., `badblocks -v /dev/sda`). * Identify I/O-bound processes using `iostat`. * Create a link to move large files/directories. 4. **Add a New Disk:** * **Simple Partition:** * Add the disk to the VM. * Verify the new disk using `df` or `lsblk`. * Use `fdisk` to create a partition (preferably LVM). * Create a filesystem, mount it, and add it to `fstab` for persistence. * **LVM Partition:** * Add the disk to the VM. * Verify with `df` or `lsblk`. * Use `fdisk` to create an LVM partition. * Set up PV, VG, and LV. * Create a filesystem, mount it, and add it to `fstab`. * **Extend LVM Partition:** * Add and create an LVM partition. * Add the new LVM partition (PV) to the existing VG. * Extend the LV and resize the filesystem. ## **Filesystem Corruption** 1. **Symptoms:** * The system fails to boot. 2. **Check Logs:** * Investigate `/var/log/messages`, `dmesg`, and other log files. * Look for bad sector logs. 3. **Run `fsck` if Bad Sectors are Found:** * Reboot the system into rescue mode (e.g., boot from CD-ROM or ISO). * Select Option 1 to mount the original root filesystem under `/mnt/sysimage`. * Edit `fstab` entries or recreate the file using `blkid`. * Reboot the system. ## **Missing or Incorrect `fstab` File** 1. **Symptoms:** * The system fails to boot. 2. **Check Logs:** * Investigate `/var/log/messages`, `dmesg`, and other log files. * Look for bad sector logs. 3. **Run `fsck` if Bad Sectors are Found:** * Reboot the system into rescue mode (e.g., boot from CD-ROM or ISO). * Select Option 1 to mount the original root filesystem under `/mnt/sysimage`. * Edit `fstab` entries or recreate the file using `blkid`. * Reboot the system. ## **Cannot `cd` to Directory (Even with Sudo Privileges)** 1. **Reasons and Resolutions:** * Directory does not exist. * Pathname conflict (relative vs absolute path). * Parent directory permission or ownership issues. * Missing executable permissions on the target directory. * Hidden directory not visible. ## **Cannot Create Links** 1. **Reasons and Resolutions:** * Target directory or file does not exist. * Pathname conflict (relative vs absolute path) — ensure the path is complete. * Parent directory permission or ownership issues. * Target file permission or ownership issues — must have read permissions. * Hidden directory or file not visible. ## **Running Out of Memory** 1. **Types of Memory:** * **Cache:** L1, L2, L3. * **RAM:** * Usage details from `free -h`: * **Total:** Total assigned memory. * **Used:** Total memory actually in use. * **Free:** Memory available for immediate use. * **Shared:** Shared memory. * **Buff/Cache:** Pages cached in memory. * **Available:** Memory that can be freed. * Check `/proc/meminfo` for detailed metrics: * File active/inactive, Anon active/inactive. * **Swap (Virtual Memory):** Monitor and manage for system stability. 2. **Resolutions:** * Identify high-memory processes using `top`, `htop`, or `ps`. * Check logs for OOM events and review memory overcommit settings in `sysctl.conf`. * Kill or restart memory-hogging processes/services. * Use `nice` to prioritize critical processes. * Add or extend swap space. * Install more physical RAM. ## **Add or Extend Swap Space** 1. **Steps to Add Swap Space:** * Create a file using `dd` to reserve disk blocks for swap. * Set file permissions to `600` and assign root ownership. * Format the file for swap with `mkswap`. * Enable swap using `swapon`. * Add the swap file to `fstab` for persistence. ## **Unable to Run Certain Commands** 1. **Troubleshooting and Resolutions:** * **Command issues:** * System-related commands may require root access. * User-defined scripts/commands might have restrictions. * **Steps to troubleshoot:** * Check permission or ownership of the command/script. * Ensure sudo privileges are configured. * Verify the absolute or relative path to the command/script. * Ensure the command is in the user's `$PATH` variable. * Confirm that the command is installed. * Check for missing or deleted command libraries. ## **System Unexpectedly Rebooting and Processes Restarting** 1. **Troubleshooting and Resolution:** * **System Reboot/Crash Reasons:** * CPU stress. * RAM stress. * Kernel fault. * Hardware fault. * **Process Restart Causes:** * System reboot triggers process restarts. * Processes might restart themselves. * Watchdog applications: * Prevent high stress on system resources. * Restart or terminate processes causing excessive stress. * **Troubleshooting Steps:** * After logging in, check system status using commands like: * `uptime`, `top`, `dmesg`, `journalctl`, `iostat -xz 1`. * Examine log files: `syslog.log`, `boot.log`, `dmesg`, `messages.log`. * Check custom application log paths. * If inaccessible, use virtual consoles (e.g., ILO, IDRAC). * Open a support case with the vendor if needed. ## **Unable to Get an IP Address** 1. **IP Assignment Methods:** * **DHCP:** * Fixed Allocation. * Dynamic Allocation. * **Static IP.** 2. **Troubleshooting Steps:** * Check network settings in the virtualization environment (e.g., VMware, VirtualBox). * Verify whether an IP address has been assigned. * Check the NIC status on the host using tools like `lspci`, `nmcli`. * Restart the network service. ## **Backup and Restore File Permissions in Linux** 1. **Backup and Restore Steps:** * The best option is to create an ACL file for directories/files before making bulk permission changes: * Backup file permissions: `getfacl -R > permissions.acl`. * Restore file permissions: `setfacl --restore=permissions.acl`. * Restore using a VM snapshot (not ideal for production environments). * Rebuild the VM (a safer option for long-term stability). ## **Useful Tips Related to Disk Partitioning** 1. **Tips for Managing Disk Partitions:** * After attaching a new disk to a VM, use `lsblk` to check its status, then rescan using: * `echo 1 > /sys/block/sda/device/rescan`. * Increasing the size of an existing disk appends additional space to the disk without affecting the existing file system or partition. * Recreating the filesystem on a block device automatically formats the old one. * For a disk with an existing partition/filesystem, share the `.vmdk` file to another VM. After mounting, the data will remain identical. # Service Level Terminology Effective service management requires understanding which behaviors matter and how to measure them. Service Level Indicators (SLIs), Service Level Objectives (SLOs), and Service Level Agreements (SLAs) help define and deliver the desired level of service. Metrics should guide appropriate actions when issues arise, ensuring the service remains healthy. *** ## **Service Level Terminology** * **SLI (Service Level Indicator):** Quantitative measures of service performance (e.g., latency, error rate, throughput, availability). * **SLO (Service Level Objective):** Target values for SLIs, specifying the desired level of service performance. * **SLA (Service Level Agreement):** Contracts with users outlining the consequences if SLOs aren’t met (e.g., financial penalties). *** ## **Service Level Indicators** * SLIs are specific metrics that indicate service health (e.g., request latency, error rate). * Common SLIs include availability (e.g., 99.9% availability = "three nines"). * Some SLIs may only be proxies for actual user experience (e.g., server-side latency vs. client-side latency). *** ## **Service Level Objectives** * SLOs set expectations for service performance and help reduce complaints. * Example: Latency SLO (e.g., average request latency < 100ms). * Choosing SLOs is complex and should reflect both user expectations and system capabilities. * Higher load often increases latency, so SLOs should account for this relationship. *** ## **Service Level Agreements** * SLAs are formal agreements between the service provider and users, typically involving penalties for unmet SLOs. * SLAs are more tied to business decisions, while SREs focus on meeting SLOs to avoid penalties. *** ## **Indicators in Practice** Focus on a handful of meaningful SLIs that matter to users, such as: * **User-facing systems:** Availability, latency, throughput. * **Storage systems:** Latency, availability, durability. * **Big data systems:** Throughput, end-to-end latency. * **All systems:** Correctness (accuracy of returned data). *** ## **Collecting and Aggregating Indicators** * Metrics can be collected server-side or client-side, depending on the aspect of user experience being measured. * Aggregating metrics (e.g., average latency) can obscure important details, such as tail latencies. * Percentiles (e.g., 99th percentile latency) offer a clearer view of performance extremes. *** ## **Best Practices** * Use percentiles rather than averages to capture the distribution of performance, especially for latency. * Standardize SLIs across services to simplify monitoring and ensure consistency. *** ## **Conclusion** * Defining and managing SLIs, SLOs, and SLAs are key to delivering a reliable service. * Prioritize metrics that matter most to users and align with your system's goals for optimal service management. # Toil ## **Toil in SRE**: * Toil refers to operational work that is manual, repetitive, automatable, tactical, and devoid of long-term value. * It is tied to running production services and scales linearly with service growth. ## Characteristics of Toil * **Manual**: Tasks that require human intervention. * **Repetitive**: Tasks done repeatedly over time. * **Automatable**: Tasks that could be done by machines. * **Tactical**: Interrupt-driven, reactive work (e.g., handling pager alerts). * **No enduring value**: Tasks that don’t result in permanent improvement. * **Scales linearly**: Effort increases with service size or usage. ## Why Reducing Toil Matters * SREs aim to keep toil under **50% of their time** to focus on long-term engineering projects. * Excessive toil leads to: * **Burnout** and **low morale**. * **Stagnation** in career growth. * **Slower progress** and productivity loss. * **Confusion** about SRE’s role as an engineering organization. * Risk of **attrition** among top engineers. ## Calculating Toil * **On-call shifts** make up a minimum of 25%-33% of an SRE’s time. * Interrupts, urgent responses, and manual processes (e.g., releases) contribute significantly to toil. ## Engineering vs. Toil * **Engineering work**: Novel, strategic, and produces lasting value. * Includes coding, creating automation tools, and system configuration. * **Overhead**: Administrative tasks like HR work or team meetings, which aren't considered toil but also don't involve engineering. ## Toil's Impact on Teams * Toil is not always bad; small amounts can be calming and provide quick wins. * However, too much toil leads to inefficiency, slower feature delivery, and lower morale. ## Conclusion * Reducing toil through automation and engineering helps scale services more efficiently and enables SREs to focus on high-value, strategic work. # Monitoring ## **Principles and Definitions** Google’s SRE teams follow essential principles for successful monitoring and alerting. These include understanding which issues merit human intervention and managing minor issues that don't require immediate attention. ### **Key Definitions:** * **Monitoring**: Collecting and displaying real-time data about system behavior, including errors, query types, and server performance. * **White-box Monitoring**: Observing internal system metrics, such as logs or JVM interfaces, for insights. * **Black-box Monitoring**: Measuring external system behavior as users experience it. * **Dashboard**: A summary view of critical service metrics, often web-based. * **Alert**: A notification, typically a page, email, or ticket, indicating a potential issue. * **Root Cause**: The underlying reason for a failure, which must be fixed to prevent recurrence. * **Node and Machine**: Refers to a single instance of a kernel in a physical or virtual environment. * **Push**: Any update to a service’s software or configuration. ## Why Monitor? Monitoring is crucial for: 1. **Long-term trend analysis**: Helps assess growth rates, like database size or user engagement. 2. **Performance comparison**: Evaluates the impact of changes, like new database software or additional infrastructure. 3. **Alerting**: Identifies current or potential problems requiring immediate or near-term attention. 4. **Dashboarding**: Provides a quick view of a system's health using key metrics like the "Four Golden Signals" (explained later). 5. **Debugging**: Retrospective analysis of unexpected changes, such as latency spikes. ### Effective Monitoring and Alerting Monitoring helps systems notify when they are broken or close to breaking. While it’s essential to alert a human for critical issues, paging can be disruptive and expensive. Too many alerts can cause alert fatigue, where important issues might be overlooked. To prevent this, alerting should be well-tuned to provide high signal with minimal noise. ### Setting Expectations for Monitoring Even with sophisticated tools, monitoring complex systems requires significant resources. Google’s SRE teams typically dedicate one or two members to maintaining these systems. The aim is to keep monitoring simple and scalable, avoiding unnecessarily complex or fragile systems. Complex dependency-based rules (e.g., alert for website issues only if the database is fine) are rare at Google due to the infrastructure's continuous evolution. Keeping monitoring rules simple and robust is essential for minimizing noise and ensuring clarity when problems arise. ### Symptoms vs. Causes Monitoring systems should differentiate between symptoms ("what’s broken") and causes ("why it’s broken"). For example, a slow website might be due to a slow database, which is a symptom for the web team but a cause for the database team. This distinction is key to effective monitoring. ### Black-box vs. White-box Monitoring Both types of monitoring are necessary. Black-box monitoring identifies immediate, user-visible problems, while white-box monitoring detects underlying causes or imminent failures by observing internal system behavior. White-box monitoring is essential for debugging and detecting masked failures, while black-box monitoring ensures alerting is focused on active, impactful issues. ### The Four Golden Signals When monitoring a user-facing system, four critical metrics should be tracked: 1. **Latency**: The time to complete a request. Both successful and failed requests should be measured. 2. **Traffic**: The demand on the system, measured in system-specific units (e.g., HTTP requests per second). 3. **Errors**: The rate of failed requests, including both explicit and policy-based failures. 4. **Saturation**: How close the system is to its full capacity, with a focus on the most constrained resources (e.g., CPU, I/O). If these four metrics are monitored and alerts are triggered when any become problematic, the system should be well covered. ### Tail Latencies and Measurement Granularity Monitoring the average performance of a system can hide issues experienced by a minority of users. To avoid this, it’s important to focus on the "tail" of the performance distribution (e.g., the slowest 1% of requests). Using latency buckets instead of averages allows for a clearer view of performance issues. Additionally, different system metrics should be measured with appropriate granularity. For instance, CPU load might need to be measured more frequently than storage availability, depending on the system’s uptime requirements. By focusing on simple, robust monitoring and keeping noise low, SRE teams can respond more effectively to system issues and minimize disruptions to both services and on-call engineers. # Release Engineering ### **Overview** * Release engineering is a discipline within software engineering focused on **building, packaging, and delivering software** in a consistent, automated, and repeatable way. * It encompasses knowledge of **source code management**, **compilers**, **build tools**, **package management**, **testing**, and **deployment processes**. * **Release Engineers:** * Collaborate with software engineers (SWEs) and Site Reliability Engineers (SREs) to define steps for software release. * Are responsible for consistent and repeatable methodologies in the release process. * Use data-driven tools to measure release velocity and other metrics. * To avoid future costly fixes, release engineering should be considered from the start of development, ensuring seamless scalability as systems grow. Early collaboration between developers, SREs, and release engineers is key to building a streamlined deployment process and preventing last-minute challenges. ### **Key Responsibilities** * **Source Code Management**: Ensuring source code is versioned properly and is ready for build and deployment. * **Build Automation**: Using tools like Jenkins, Gradle, or Bazel to automate the compilation and packaging of software. * **Testing Integration**: Ensuring that tests are run automatically during the build process to catch issues early. * **Package Management**: Managing software dependencies and ensuring consistency in different environments through package managers like Maven, npm, or Docker. ### **Deployment Strategies** * **Canary releases**: Gradual deployment of a new release to a subset of users or systems to verify the stability of the release before full deployment. * **Blue-Green deployments**: A strategy where two identical production environments (blue and green) are used to switch traffic from one environment to another, enabling minimal downtime. * **Rollbacks**: If an issue is detected in a release, the system should support rolling back to the previous stable version seamlessly. ### **Configuration Management** * Ensures that configuration files and settings are versioned and consistent across different environments. * Configurations should either be decoupled from the code or bundled with the build to ensure consistency. ### **Security and Access Control** * Security measures and role-based access control are essential to regulate who can perform specific actions during the build and release process, such as approving code changes or initiating a deployment. ### **Continuous Deployment Tools** * Tools like **GitHub Actions, GitLab CI, Google Cloud Build,** **Jenkins**, **Travis CI and** **CircleCI** help automate the release pipeline by managing builds, tests, and deployments. ### **Challenges** * **Scaling** release processes across multiple teams and projects. * Managing the complexity of dependencies, especially in **microservices** environments. * Handling legacy systems while adopting modern release engineering practices. # Best Practices ## **Stability vs. Productivity** * **Goal:** Balance system **stability** with developer **productivity**. * **Exploratory coding:** Temporary code for problem exploration; not meant for production. * **Production systems:** Prioritize stability, but reliable processes should also **enhance productivity**. * **Faster bug detection and fixes** in reliable systems. * Focus on system **performance** and **functionality**. *** ## **Boring = Good** * In software, "boring" systems are **predictable** and stable. * **Essential complexity:** Unavoidable complexity inherent to a problem. * **Accidental complexity:** Extra complexity introduced by poor design choices (e.g., Java's garbage collection in a web server). * **SRE's job:** Minimize accidental complexity. *** ## **Eliminate Dead Code** * **Unused code = Liability** * Creates confusion and increases the risk of bugs. * **Negative lines of code:** Removing unnecessary code can be more valuable than adding features. * Smaller codebase = **Easier to maintain and understand**. *** ## **Minimal APIs** * **Fewer methods and arguments = Easier to use and maintain.** * Simplicity in API design improves focus on core functionality. * **Antoine de Saint-Exupery quote:** "Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away." *** ## **Modularity** * **Loose coupling = Flexibility and scalability.** * Independent updates to system components minimize disruption. * Apply to distributed systems and APIs. * **Versioning APIs** allows for smooth upgrades without breaking dependencies. * Modular systems help manage growing complexity. *** ## **Simple Releases** * **Releases should be incremental**—small batches, not big ones. * Easier to test and troubleshoot. * If many changes are released together, it’s harder to track down the cause of any issues. * Incremental releases = more control, faster progress. *** ## **Key Points to Remember** * **Simplicity = Reliability**: Keeping things simple helps ensure systems are more reliable. * **Boring is good**: Predictable, stable systems are the goal. * **Focus on clarity**: Eliminate unnecessary code and complexity. * **Minimalism in design**: Smaller APIs and modular systems are easier to maintain and scale. # On-Call ## **Purpose of On-Call** * **Goal**: Be available and respond to production incidents with appropriate urgency. * **Responsibilities**: * Diagnose, mitigate, fix, or escalate incidents. * Perform non-urgent production tasks when not handling incidents. *** ## **Key Considerations for On-Call** * **Balance**: Engineers should not sacrifice health for reliability. * **Goal**: SREs should handle a healthy mix of project work and on-call duties (50% project work is ideal). * **Psychological Safety**: Support systems and escalation procedures must exist to reduce stress. * **Compensation**: Time-off or cash incentives for on-call work to prevent burnout. *** ## **Example On-Call Setups** * **Google’s Approach**: * **Training Roadmap**: Checklist of focus areas (e.g., monitoring systems, debugging, handling traffic). * **Deep Dives**: Focus on gaining expertise in services. * **Onboarding**: Mentoring, shadowing on-call shifts, and practice with “Wheel of Misfortune” disaster exercises. * **Playbooks**: Documentation to handle alerts and reduce Mean Time to Repair (MTTR). * **Evernote’s Cloud Migration**: * **Reframing Alerts**: Shift focus from infrastructure-level events to user impact (e.g., API responsiveness). * **On-Call Rotation**: Classify events into three categories: * **P1**: Immediate action needed, page on-call. * **P2**: Handle next business day, send email. * **P3**: Informational only. * **Post-Incident Review**: Every P1 and P2 event has a postmortem and continuous improvement cycle. *** ## **Managing Pager Load** * **Definition**: Number of paging incidents during a shift. * **Techniques to Manage Pager Load**: * Ensure only actionable issues trigger pages. * Use automation for routine fixes or escalate lower-priority issues to tickets. * Focus on **high signal-to-noise ratio** for alerts. * **Appropriate Response Times**: * **Critical (5 min)**: Requires immediate action (e.g., revenue-impacting outages). * **Moderate (30 min)**: Less critical issues. * **Low (during work hours)**: Non-urgent tasks like pre-launch backups. *** ## **Reducing Pager Load** * **Identifying Causes**: * Existing production bugs. * New bugs introduced into production. * Alerting thresholds and human processes. * **Strategies**: * **Fix existing bugs** before releasing new features. * **Automate** production changes to reduce human error. * **Rollback strategy**: Detect, rollback, fix, and reintroduce fixes. *** ## **Best Practices for Alerts** * Alerts should be **immediately actionable** and not overwhelm engineers. * **SLO-based alerts**: Only page when error budget is burned. * **Team Review**: Alerts should be thoroughly reviewed and tested in production before implementation. *** ## **Rigor of Follow-Up** * **Root Cause Analysis**: * Identify and prevent repeat incidents. * Focus on improving systems, not just patching immediate bugs. * **Systemic Fixes**: * Aim for systemic improvements (e.g., automation, better monitoring). *** ## **On-Call Flexibility** * **Shift Length**: 12-hour shifts are preferred for sustainable on-call work. * **Flexibility**: Accommodate personal life changes (e.g., part-time, temporary breaks). * **Automated Scheduling**: Use automated tools to handle complex schedules fairly. *** ## **Team Dynamics** * **Building Positive Relations**: * Encourage team bonding (e.g., offsite activities, lunches). * Co-locate on-call teams to foster collaboration. * **Empowerment**: * **SRE Ownership**: Make SREs responsible for site reliability, working alongside feature developers. *** ## **Conclusion** * On-call is critical to site reliability but must be structured to prevent burnout and foster team collaboration. * Systematic improvement in on-call processes and team dynamics leads to healthier, more effective teams. # Alerting ## **Purpose of Alerting** * **Goal**: Turn SLOs (Service Level Objectives) into actionable alerts. * **Benefit**: Respond to issues before consuming too much of the error budget. *** ## **Alerting Considerations** * **Precision**: Detect significant events without false alerts. * **Recall**: Ensure all significant events trigger alerts. * **Detection Time**: Minimize the time taken to detect issues. * **Reset Time**: Ensure alerts resolve quickly once issues are fixed. *** ## **Six Approaches to Alerting** **1. Target Error Rate ≥ SLO Threshold** * **Implementation**: Trigger alerts if the error rate exceeds SLO for a short window (e.g., 10 minutes). * **Pros**: Simple, quick detection. * **Cons**: Poor precision—too many false positives for minor issues. **2. Increased Alert Window** * **Implementation**: Increase the alert window (e.g., 36 hours) to improve precision. * **Pros**: Better precision than short windows. * **Cons**: Long reset times and higher memory costs. **3. Incrementing Alert Duration** * **Implementation**: Only trigger alerts if the error rate stays above the threshold for a set duration. * **Pros**: Better precision for sustained issues. * **Cons**: Poor recall and slow detection time for severe issues. **4. Alert on Burn Rate** * **Definition**: Burn rate = how fast the service consumes the error budget. * **Implementation**: Alert when the burn rate exceeds a critical threshold (e.g., burn rate of 36). * **Pros**: Good precision and detection time. * **Cons**: May miss lower, slow-moving errors. **5. Multiple Burn Rate Alerts** * **Implementation**: Use multiple burn rates for different error severities (e.g., 2% budget in 1 hour, 5% in 6 hours). * **Pros**: Adaptive alerting for both fast and slow errors; prioritizes alerts based on severity. * **Cons**: More complex to manage multiple burn rates and windows. **6. Multiwindow, Multi-Burn-Rate Alerts (Recommended)** * **Implementation**: Combine short and long windows with multiple burn rates (e.g., alert if both 1-hour and 5-minute windows exceed burn rate thresholds). * **Pros**: Best approach for managing precision, recall, detection, and reset times; reduces false positives. *** ## **Handling Low-Traffic Services** * **Problem**: High sensitivity to errors in low-traffic services causes false alerts. * **Solutions**: * **Generate artificial traffic** to simulate user activity. * **Combine smaller services** into a single alerting system. * **Modify product design** to reduce impact of individual failed requests. *** ## **Handling Extreme Availability Goals** * **Low Availability**: E.g., 90% availability—errors may go unnoticed in long error budgets. * **High Availability**: E.g., 99.999% availability—100% outages can deplete error budgets in seconds. * **Solution**: Design systems to avoid 100% outages or implement gradual rollouts (e.g., canarying). *** ## **Scalable Alerting Framework** * **Avoid Custom Parameters**: Don't specify alert parameters for every microservice. * **Group Requests into Buckets**: * **Critical**: E.g., login requests (99.99% availability). * **High Priority**: E.g., user interaction (99.9% availability). * **Low Priority**: Non-urgent requests with minimal user impact. *** ## **Conclusion** * **Best Strategy**: Multiwindow, multi-burn-rate alerting is the most reliable way to defend SLOs. * **Objective**: Set alerts to notify for actionable, significant events that impact the error budget. # Containers ### **What** * Lightweight, portable, and isolated environments for applications. * Package applications and their dependencies for consistent deployment. * Encapsulate software code, runtime, system tools, libraries, and settings. ### **Why** * Ensure applications run the same across different environments. * Share the host OS kernel, making them more efficient than traditional virtual machines. * Supports microservices architectures. * Simplifies application deployment. * Improves scalability. * Enhances DevOps practices by streamlining the development-to-production pipeline. * Enables more efficient resource utilization. ### **Popular Platforms** * **Docker**: Tools for creating, distributing, and running containers. # Docker > Docker is an open platform for developing, shipping, and running applications. Docker enables you to separate your applications from your infrastructure so you can deliver software quickly. With Docker, you can manage your infrastructure in the same ways you manage your applications. By taking advantage of Docker’s methodologies for shipping, testing, and deploying code quickly, you can significantly reduce the delay between writing code and running it in production. ## Why * Faster development by standardizing environments * Uses less resources than VMs * Deployments are more responsive # Best Practices The following development patterns have proven to be helpful for people building applications with Docker. # Image Building https\://docs.docker.com/get-started/09\_image\_best/ ## Security scanning When you have built an image, it is a good practice to scan it for security vulnerabilities using the `docker scan` command. Docker has partnered with [Snyk](https://snyk.io/) to provide the vulnerability scanning service. > **Note** > > You must be logged in to Docker Hub to scan your images. Run the command `docker scan --login`, and then scan your images using`docker scan `. For example, to scan the `getting-started` image you created earlier in the tutorial, you can just type ``` $ docker scan getting-started ``` The scan uses a constantly updated database of vulnerabilities, so the output you see will vary as new vulnerabilities are discovered, but it might look something like this: ``` ✗ Low severity vulnerability found in freetype/freetype Description: CVE-2020-15999 Info: https://snyk.io/vuln/SNYK-ALPINE310-FREETYPE-1019641 Introduced through: freetype/freetype@2.10.0-r0, gd/libgd@2.2.5-r2 From: freetype/freetype@2.10.0-r0 From: gd/libgd@2.2.5-r2 > freetype/freetype@2.10.0-r0 Fixed in: 2.10.0-r1 ✗ Medium severity vulnerability found in libxml2/libxml2 Description: Out-of-bounds Read Info: https://snyk.io/vuln/SNYK-ALPINE310-LIBXML2-674791 Introduced through: libxml2/libxml2@2.9.9-r3, libxslt/libxslt@1.1.33-r3, nginx-module-xslt/nginx-module-xslt@1.17.9-r1 From: libxml2/libxml2@2.9.9-r3 From: libxslt/libxslt@1.1.33-r3 > libxml2/libxml2@2.9.9-r3 From: nginx-module-xslt/nginx-module-xslt@1.17.9-r1 > libxml2/libxml2@2.9.9-r3 Fixed in: 2.9.9-r4 ``` The output lists the type of vulnerability, a URL to learn more, and importantly which version of the relevant library fixes the vulnerability. There are several other options, which you can read about in the [docker scan documentation](https://docs.docker.com/engine/scan/). As well as scanning your newly built image on the command line, you can also [configure Docker Hub](https://docs.docker.com/docker-hub/vulnerability-scanning/) to scan all newly pushed images automatically, and you can then see the results in both Docker Hub and Docker Desktop. ![Hub vulnerability scanning](https://docs.docker.com/get-started/images/hvs.png) ## Image layering Did you know that you can look at what makes up an image? Using the `docker image history` command, you can see the command that was used to create each layer within an image. 1. Use the `docker image history` command to see the layers in the `getting-started` image you created earlier in the tutorial. ``` $ docker image history getting-started ``` You should get output that looks something like this (dates/IDs may be different). ``` IMAGE CREATED CREATED BY SIZE COMMENT a78a40cbf866 18 seconds ago /bin/sh -c #(nop) CMD ["node" "src/index.j… 0B f1d1808565d6 19 seconds ago /bin/sh -c yarn install --production 85.4MB a2c054d14948 36 seconds ago /bin/sh -c #(nop) COPY dir:5dc710ad87c789593… 198kB 9577ae713121 37 seconds ago /bin/sh -c #(nop) WORKDIR /app 0B b95baba1cfdb 13 days ago /bin/sh -c #(nop) CMD ["node"] 0B 13 days ago /bin/sh -c #(nop) ENTRYPOINT ["docker-entry… 0B 13 days ago /bin/sh -c #(nop) COPY file:238737301d473041… 116B 13 days ago /bin/sh -c apk add --no-cache --virtual .bui… 5.35MB 13 days ago /bin/sh -c #(nop) ENV YARN_VERSION=1.21.1 0B 13 days ago /bin/sh -c addgroup -g 1000 node && addu… 74.3MB 13 days ago /bin/sh -c #(nop) ENV NODE_VERSION=12.14.1 0B 13 days ago /bin/sh -c #(nop) CMD ["/bin/sh"] 0B 13 days ago /bin/sh -c #(nop) ADD file:e69d441d729412d24… 5.59MB ``` Each of the lines represents a layer in the image. The display here shows the base at the bottom with the newest layer at the top. Using this, you can also quickly see the size of each layer, helping diagnose large images. 2. You’ll notice that several of the lines are truncated. If you add the `--no-trunc` flag, you’ll get the full output (yes... funny how you use a truncated flag to get untruncated output, huh?) ``` $ docker image history --no-trunc getting-started ``` ## Layer caching Now that you’ve seen the layering in action, there’s an important lesson to learn to help decrease build times for your container images. > Once a layer changes, all downstream layers have to be recreated as well Let’s look at the Dockerfile we were using one more time... ``` # syntax=docker/dockerfile:1 FROM node:12-alpine RUN apk add --no-cache python2 g++ make WORKDIR /app COPY . . RUN yarn install --production CMD ["node", "src/index.js"] ``` Going back to the image history output, we see that each command in the Dockerfile becomes a new layer in the image. You might remember that when we made a change to the image, the yarn dependencies had to be reinstalled. Is there a way to fix this? It doesn’t make much sense to ship around the same dependencies every time we build, right? To fix this, we need to restructure our Dockerfile to help support the caching of the dependencies. For Node-based applications, those dependencies are defined in the `package.json` file. So, what if we copied only that file in first, install the dependencies, and *then* copy in everything else? Then, we only recreate the yarn dependencies if there was a change to the `package.json`. Make sense? 1. Update the Dockerfile to copy in the `package.json` first, install dependencies, and then copy everything else in. ``` # syntax=docker/dockerfile:1 FROM node:12-alpine RUN apk add --no-cache python2 g++ make WORKDIR /app COPY package.json yarn.lock ./ RUN yarn install --production COPY . . CMD ["node", "src/index.js"] ``` 2. Create a file named `.dockerignore` in the same folder as the Dockerfile with the following contents. ```ignore node_modules ``` `.dockerignore` files are an easy way to selectively copy only image relevant files. You can read more about this [here](https://docs.docker.com/engine/reference/builder/#dockerignore-file). In this case, the `node_modules` folder should be omitted in the second `COPY` step because otherwise, it would possibly overwrite files which were created by the command in the `RUN` step. For further details on why this is recommended for Node.js applications and other best practices, have a look at their guide on [Dockerizing a Node.js web app](https://nodejs.org/en/docs/guides/nodejs-docker-webapp/). 3. Build a new image using `docker build`. ``` $ docker build -t getting-started . ``` You should see output like this... ``` Sending build context to Docker daemon 219.1kB Step 1/6 : FROM node:12-alpine ---> b0dc3a5e5e9e Step 2/6 : WORKDIR /app ---> Using cache ---> 9577ae713121 Step 3/6 : COPY package.json yarn.lock ./ ---> bd5306f49fc8 Step 4/6 : RUN yarn install --production ---> Running in d53a06c9e4c2 yarn install v1.17.3 [1/4] Resolving packages... [2/4] Fetching packages... info fsevents@1.2.9: The platform "linux" is incompatible with this module. info "fsevents@1.2.9" is an optional dependency and failed compatibility check. Excluding it from installation. [3/4] Linking dependencies... [4/4] Building fresh packages... Done in 10.89s. Removing intermediate container d53a06c9e4c2 ---> 4e68fbc2d704 Step 5/6 : COPY . . ---> a239a11f68d8 Step 6/6 : CMD ["node", "src/index.js"] ---> Running in 49999f68df8f Removing intermediate container 49999f68df8f ---> e709c03bc597 Successfully built e709c03bc597 Successfully tagged getting-started:latest ``` You’ll see that all layers were rebuilt. Perfectly fine since we changed the Dockerfile quite a bit. 4. Now, make a change to the `src/static/index.html` file (like change the `` to say “The Awesome Todo App”). 5. Build the Docker image now using `docker build -t getting-started .` again. This time, your output should look a little different. ``` Sending build context to Docker daemon 219.1kB Step 1/6 : FROM node:12-alpine ---> b0dc3a5e5e9e Step 2/6 : WORKDIR /app ---> Using cache ---> 9577ae713121 Step 3/6 : COPY package.json yarn.lock ./ ---> Using cache ---> bd5306f49fc8 Step 4/6 : RUN yarn install --production ---> Using cache ---> 4e68fbc2d704 Step 5/6 : COPY . . ---> cccde25a3d9a Step 6/6 : CMD ["node", "src/index.js"] ---> Running in 2be75662c150 Removing intermediate container 2be75662c150 ---> 458e5c6f080c Successfully built 458e5c6f080c Successfully tagged getting-started:latest ``` First off, you should notice that the build was MUCH faster! And, you’ll see that steps 1-4 all have `Using cache`. So, hooray! We’re using the build cache. Pushing and pulling this image and updates to it will be much faster as well. Hooray! ## Multi-stage builds While we’re not going to dive into it too much in this tutorial, multi-stage builds are an incredibly powerful tool to help use multiple stages to create an image. There are several advantages for them: * Separate build-time dependencies from runtime dependencies * Reduce overall image size by shipping *only* what your app needs to run ### Maven/Tomcat example When building Java-based applications, a JDK is needed to compile the source code to Java bytecode. However, that JDK isn’t needed in production. Also, you might be using tools like Maven or Gradle to help build the app. Those also aren’t needed in our final image. Multi-stage builds help. ``` # syntax=docker/dockerfile:1 FROM maven AS build WORKDIR /app COPY . . RUN mvn package FROM tomcat COPY --from=build /app/target/file.war /usr/local/tomcat/webapps ``` In this example, we use one stage (called `build`) to perform the actual Java build using Maven. In the second stage (starting at `FROM tomcat`), we copy in files from the `build` stage. The final image is only the last stage being created (which can be overridden using the `--target` flag). ### React example When building React applications, we need a Node environment to compile the JS code (typically JSX), SASS stylesheets, and more into static HTML, JS, and CSS. If we aren’t doing server-side rendering, we don’t even need a Node environment for our production build. Why not ship the static resources in a static nginx container? ``` # syntax=docker/dockerfile:1 FROM node:12 AS build WORKDIR /app COPY package* yarn.lock ./ RUN yarn install COPY public ./public COPY src ./src RUN yarn run build FROM nginx:alpine COPY --from=build /app/build /usr/share/nginx/html ``` Here, we are using a `node:12` image to perform the build (maximizing layer caching) and then copying the output into an nginx container. Cool, huh? # Docker Development ## How to keep your images small Small images are faster to pull over the network and faster to load into memory when starting containers or services. There are a few rules of thumb to keep image size small: * Start with an appropriate base image. For instance, if you need a JDK, consider basing your image on the official `openjdk` image, rather than starting with a generic `ubuntu` image and installing `openjdk` as part of the Dockerfile. * [Use multistage builds](https://docs.docker.com/build/building/multi-stage/). For instance, you can use the `maven` image to build your Java application, then reset to the `tomcat` image and copy the Java artifacts into the correct location to deploy your app, all in the same Dockerfile. This means that your final image doesn’t include all of the libraries and dependencies pulled in by the build, but only the artifacts and the environment needed to run them. * If you need to use a version of Docker that does not include multistage builds, try to reduce the number of layers in your image by minimizing the number of separate `RUN` commands in your Dockerfile. You can do this by consolidating multiple commands into a single `RUN` line and using your shell’s mechanisms to combine them together. Consider the following two fragments. The first creates two layers in the image, while the second only creates one. ``` RUN apt-get -y update RUN apt-get install -y python ``` ``` RUN apt-get -y update && apt-get install -y python ``` * If you have multiple images with a lot in common, consider creating your own [base image](https://docs.docker.com/develop/develop-images/baseimages/) with the shared components, and basing your unique images on that. Docker only needs to load the common layers once, and they are cached. This means that your derivative images use memory on the Docker host more efficiently and load more quickly. * To keep your production image lean but allow for debugging, consider using the production image as the base image for the debug image. Additional testing or debugging tooling can be added on top of the production image. * When building images, always tag them with useful tags which codify version information, intended destination (`prod` or `test`, for instance), stability, or other information that is useful when deploying the application in different environments. Do not rely on the automatically-created `latest` tag. ## Where and how to persist application data * **Avoid** storing application data in your container’s writable layer using [storage drivers](https://docs.docker.com/storage/storagedriver/select-storage-driver/). This increases the size of your container and is less efficient from an I/O perspective than using volumes or bind mounts. * Instead, store data using [volumes](https://docs.docker.com/storage/volumes/). * One case where it is appropriate to use [bind mounts](https://docs.docker.com/storage/bind-mounts/) is during development, when you may want to mount your source directory or a binary you just built into your container. For production, use a volume instead, mounting it into the same location as you mounted a bind mount during development. * For production, use [secrets](https://docs.docker.com/engine/swarm/secrets/) to store sensitive application data used by services, and use [configs](https://docs.docker.com/engine/swarm/configs/) for non-sensitive data such as configuration files. If you currently use standalone containers, consider migrating to use single-replica services, so that you can take advantage of these service-only features. # CLI Cheat Sheet <figure><img src="/files/n7jY6dPMGdDvXRJqmhNc" alt=""><figcaption></figcaption></figure> # Container Orchestration # Kubernetes Kubernetes is an open source, portable platform designed for managing containerized workloads and services through declarative configuration and automation. The term "Kubernetes" comes from the Greek word for helmsman or pilot, and the abbreviation "K8s" stems from counting the eight letters between "K" and "s." Originating as a Google project in 2014, Kubernetes integrates more than 15 years of Google's expertise in handling large-scale production workloads with community-driven best practices. <figure><img src="/files/IHwywrPT7Tqd4fU4UI09" alt="" width="563"><figcaption></figcaption></figure> ## History ### **Traditional deployment era** Early on, organizations ran applications on physical servers and there was no way to define resource boundaries for applications. This caused issues where one application would take up most of the resources. ### **Virtualized deployment era** As a solution, virtualization was introduced. It allowed better utilization of resources in a physical server and security. ### **Container deployment era** Containers are comparable to virtual machines but with relaxed isolation properties, shared operating system, making them lightweight and portable across clouds and OS distributions, gaining popularity for agile application creation, continuous development, DevOps separation, observability, environmental consistency, cloud and OS distribution portability, application-centric management, and resource efficiency. <figure><img src="/files/HDigXge77XFp97L4Ygc5" alt="" width="563"><figcaption>History Visualized</figcaption></figure> \ # Benefits **Service Discovery and Load Balancing:** * Expose containers using DNS names or IP addresses. * Built-in load balancing for stable deployment. **Storage Orchestration:** * Automatically mount various storage systems (local storage, public cloud providers). **Automated Rollouts and Rollbacks:** * Define and automate the desired state of deployed containers. * Facilitate controlled updates and rollbacks. **Automatic Bin Packing:** * Efficiently allocate CPU and memory resources. * Optimize container placement on a cluster of nodes. **Self-Healing:** * Kubernetes restarts, replaces, and manages containers in response to failures. * Ensure continuous service availability. **Secret and Configuration Management:** * Safely store and update sensitive information. * Without rebuilding container images or exposing secrets. **Batch Execution:** * Handle batch and CI workloads. * Replace failed containers as needed. **Horizontal Scaling:** * Easily scale applications up or down based on demand. * Manual, UI-based, or automated CPU usage-triggered scaling. **IPv4/IPv6 Dual-Stack:** * Allocate both IPv4 and IPv6 addresses to Pods and Services. * Comprehensive networking support. **Designed for Extensibility:** * Extend Kubernetes clusters with additional features. * Without modifying upstream source code. # Cheat Sheet {% file src="/files/qq3hr6lyMO4PgeTq64V4" %} # Components A Kubernetes **cluster** is a **set of nodes** that run containerized applications. It comes with something called a **control plane** that manages the overall state of the cluster, as well as make global decisions about the cluster (for example, scheduling). It is responsible for maintaining the desired state of the cluster, responding to user requests, and managing the deployment and scaling of applications. ## Control Plane Components <figure><img src="/files/IsTODGQw0WSHvYaOcgZm" alt=""><figcaption><a href="https://kubesphere.io/blogs/monitoring-k8s-control-plane/">Reference</a></figcaption></figure> ### **API Server** * The API server is a component that exposes the Kubernetes API. It serves as the front end for the control plane. Users, other components, and external tools communicate with the cluster through the API server. ### **etcd** * etcd is an open source distributed key-value store that stores the configuration data of the cluster, representing the overall state of the system. The control plane components watch for changes in etcd and react accordingly to maintain the desired state. ### **Controller Manager** * The controller manager is responsible for running controller processes that regulate the state of the cluster. Examples include the Replication Controller, which ensures the correct number of replicas for a set of pods, and the Node Controller, which monitors and responds to changes in the nodes of the cluster. ### **Scheduler** * The scheduler is responsible for placing pods onto nodes in the cluster. It takes into account factors such as resource requirements, hardware constraints, and affinity/anti-affinity specifications when making placement decisions. These components work together to ensure that the cluster is in the desired state, and they continuously monitor and adjust the cluster to maintain that state. The control plane components are typically distributed across multiple nodes for high availability and fault tolerance. ## Worker Node Components <figure><img src="/files/rjyckqMEXwjFjUUiDGBr" alt=""><figcaption><a href="https://itnext.io/one-story-for-effortlessly-passing-kubernetes-interview-questions-in-2023-f93e828cee9f">Reference</a></figcaption></figure> The worker nodes are responsible for hosting the application pods in the cluster. ### **Kubelet** * Kubelet is an agent that runs on each node and is responsible for creating the pods by the provided YAML specs, reporting the health status of the pods and providing status information on the node. (network, disk space, etc.) ### **Container Runtime** * The container runtime is the software responsible for running containers. Kubernetes supports various container runtimes, including Docker, containerd, and others. The container runtime is responsible for pulling container images from a container registry, creating containers, and managing their lifecycle. ### **Kube-proxy** * Kube-proxy is responsible for network proxying on the worker nodes. It maintains network rules to allow communication to Pods from network sessions inside or outside of the cluster. Kube-proxy enables the communication between different Pods and services in the cluster. # Pods **Pods** are the smallest deployable units of computing that you can create and manage in Kubernetes. A Pod is similar to a set of containers with shared namespaces and shared filesystem volumes. <figure><img src="/files/UgzSjIAKw546Mb8dAaxF" alt=""><figcaption></figcaption></figure> **Init containers** are executed before both the sidecar and main application containers, and their successful completion is a prerequisite for the other containers to start. Init containers offer versatility, allowing actions like checking for application dependencies or configuring the environment for main and sidecar containers. **Sidecar containers** run concurrently with the main application container and serve various purposes. For instance, in Istio, a sidecar container functions as a traffic proxy for the main container, and it can also handle tasks such as logging and monitoring. # Workload Resources In Kubernetes, applications run in containers within Pods. To simplify management, higher-level workload objects are created using the Kubernetes API. Key workload types include the Deployment, the StatefulSet and the DaemonSet. <figure><img src="/files/BzzYuNLcyn0Oclr98J15" alt=""><figcaption></figcaption></figure> **Deployment** is the go-to resource for easily deploying stateless applications in a Kubernetes cluster. It uses a ReplicaSet for straightforward version rollbacks. The pod naming convention is `<deployment-name>`-`<replicaset-id>`-`<pod-id>`. **StatefulSet**, introduced in Kubernetes 1.9, is designed for stable hosting of stateful applications. It independently manages pods and requires a headless service for network identification and DNS resolution. Pods in a StatefulSet have unique names like `<Statefulset-name>`-0, `<Statefulset-name>`-1, and each has its own persistence volume claim. StatefulSets are ideal for stateful applications, such as databases, where replica identification and graceful upgrades are essential. # Best Practices <https://aws.github.io/aws-eks-best-practices/> # Developer Portal 👨‍💻 ## Introduction In early 2023, our engineering teams were spending one to three weeks per month manually setting up new projects leading to significant productivity loss and inconsistent configurations. After doing a developer survey and quantifying the impact, I built a self-service developer portal using Backstage that reduced project setup time from one week to less than an hour. This presentation walks through the journey of designing and implementing this solution. ## Problem Space 🔍 * ⏰ **Time Cost**: \~1 week per project setup * 📈 **Monthly Impact**: 1-3 projects = Up to 3 weeks engineering time lost * 👥 **Scale**: Supporting 200+ engineers * 💸 **Business Cost**: \~$15,000 per project setup (assuming average engineer salary) #### Time Breakdown ⏱️ ``` Project Setup Timeline: ├── Infrastructure Setup: 2-3 days ├── Access Management: 1 day ├── Platform Integration: 1-2 days └── Testing & Validation: 1 day ``` ## Pain Points Identified Through Survey <figure><img src="/files/YvJMmUbqq8IIwogGvHEy" alt=""><figcaption>Manual Project Setup Flow</figcaption></figure> * **Infrastructure Challenges** 🏗️ * Engineers lacking IaC knowledge and taking time to understand guidelines * Limited terraform expertise in product teams * **Repository Setup Issues** 🔐 * Manual repository creation * Time-consuming GitHub team setup * Inconsistent repository settings * **Platform Integration Friction** 🔄 * Context switching between platforms: * 🔍 Sentry for error tracking * 📊 Datadog for monitoring * 🚦 LaunchDarkly for feature flags * [️🚨](https://emojis.wiki/security/) Snyk for security scanning * Manual resource creation in each platform # Solution Overview 🎯 ## Vision Build a self-service developer portal that automates project creation and standardizes infrastructure setup across the organization. ## Core Requirements ### **Functional Requirements** * **Project Creation** * Create new projects from templates * Configure project CI/CD and infrastructure * Automate repository setup and permissions * Set up required platform integrations (Sentry, Datadog, etc.) * **Platform Features** * Centralized API documentation * Service discovery ### **Non-functional Requirements** * **Performance** * Reduce project setup time from 1 week to under 2 hours * Handle concurrent project creation requests * **Scalability** * Support 200+ engineers * Handle multiple project templates * Scale across different engineering teams * **Security** * Enterprise SSO Integration * Role based access control * **Maintainability** * Maintainable template system * Standardized infrastructure patterns * Clear documentation * **Reliability** * Consistent project creation * Error handling and recovery ## Why Backstage? * Wrote a tech brief that included a decision matrix with the popular Developer Portal platforms * Backstage emerged as the optimal choice based on: * Open source and free * Great documentation * Intuitive UI/UX * 100+ existing plugins * Strong developer experience focus {% embed url="<https://youtu.be/85TQEpNCaU0>" %} # System Architecture 🏗️ Intro ## Infrastructure Overview In this section, I will delve into an example implementation of Backstage using a combination of Google Cloud Platform (GCP) services, Okta for Single Sign-On (SSO) authentication and authorization, and GitHub integration to access organization data. Here's a preliminary glimpse into the system's structure: <figure><img src="/files/QvwE0llebILCkO5S4DBq" alt=""><figcaption></figcaption></figure> * **Backstage Layer** * Backstage deployed on Cloud Run * Traffic outside of work hours is very low to none * ✅ Cost savings from going serverless and scaling to 0 instances when not in use * ⚠️ Tradeoff was cold starts. The first request needs to wait for a new container, causing a delay (10-15s). * Cloud Run has [autoscaling](https://cloud.google.com/run/docs/about-instance-autoscaling), meaning that you don't necessarily need to put a load balancer in front! * **Storage Layer** * Cloud SQL: User, project and organization metadata * Cloud Storage: API documentation stored (swagger/openAPI) * Secret Manager: App secrets fetched and used as environment variables during build and runtime * **Integration Layer** * Okta: Authentication/Authorization * GitHub: Repository and organization data management * Third-party tools: Monitoring via Cloud Logs and Google Analytics. * **Nice-to-Have (But Not Achieved)** * Leveraging [cache](https://backstage.io/docs/overview/architecture-overview/#cache) stores (Redis, etc.) to improve performance ### CI/CD Deployment Pipeline 🔄 <figure><img src="/files/Xel4xINcJQSZWWfZmTrd" alt=""><figcaption></figcaption></figure> **Pipeline Components** * **Trigger**: Cloud Build trigger listening for merge to default branch events * **Build Process**: * Pull latest Backstage image from Artifact Registry (build cache) * Build new Docker image with latest changes * Push updated image to Artifact Registry * Deploy new instance to Cloud Run * **Nice-to-Have (But Not Achieved):** * Staging environment * Canary rollout * Small percentage -> monitor metrics -> increase traffic if stable **Key Benefits** * **Efficient Builds**: Docker layer caching reduces build time * **GitOps Deployment**: Zero-touch deployment process * **Rollback Capability**: Easy rollback to previous versions # Implementation Journey 🛠️ ## Phase 0: POC (Weeks 1 & 2) * **Goals** * Local Backstage setup * Quick example template to showcase automation capabilities * Service Discovery example * read Org data from Github * Scaffolder Automation example * write to Github (pull requests & repository creation) * **Challenges** * Learning about platform and configuration * Getting permissions to create a Github App * **Outcomes** * Transferred knowledge to team * Demo'd platform capabilities * Received approval to implement ## Phase 1: Deployment, Service Discovery & API Docs (Quarter 1) * **Goals** * Deploy Backstage to Cloud * Authentication via Okta SSO integration * API Documentation Feature Setup * **Challenges** * IaC and CI/CD setup * Multiple meetings with Cloud Engineering team to figure out right architecture * Settled on Cloud Run for cost savings * GitOps CI/CD Setup * Deployments on merge to default branch * Automated tests (linting, unit, e2e, etc.) on push to any branch * Authentication requirements gathering and implementation * Worked with IT to receive secrets and setup Okta configuration * API documentation publishing feature released * Friction to push teams to use feature * Cross-team alignment on service discovery standards * Once again friction to get teams to implement across org repositories * **Outcomes** * First few Backstage deployments * Secure authentication flow * CI/CD pipeline setup * Centralized API documentation started * Found out it takes time to get adoption :crying\_cat\_face: * Initial developer feedback ## Phase 2: Scaffolder Templates (Quarter 2) * **Goals** * Achieve full API documentation adoption * Scaffolder templates framework * Templates for backend and frontend engineers * Plugins to reduce context switching * Google Analytics to track usage of Backstage * **Challenges** * Gathering unwritten engineering standards * Working with architecture team and engineers to document best practices and create re-usable "skeleton" repositories * Long hours spent testing and debugging templates * Implementing plugins for various patforms (datadog, sentry, snyk and entity feedback) * **Outcomes** * Crentralized all of our API docs * Added two backend templates (nodejs/php microservice) and one frontend template (vuejs microfrontend) * Documented engineering standards for all future projects * Quick wins implementing plugins that reduced context switching and allowed engineers to leave feedback on eachother's projects ## Phase 3: More Templates & Enhancements (Quarter 3-4) * **Goals** * Work with new teams to create more templates * Implement cost insights * Automate engineering standard checks against entities and visualization * **Challenges** * Org wide tagging of cloud resources to allow getting GCP billing data linked to Backstage entities * Took a lot of effort because it affected lots of team's projects IaC * Migrating engineering standards documentation to automated checks using Spotify's Sound Check plugin * Lots of time spent figuring out how to get the correct metadata to make sure checks are reliable * **Outcomes** * Multiple team-specific and generic templates like pub/sub topics * Users can visualize cloud costs per entity <figure><img src="/files/wdCZiz1rXwKHM08xpLHu" alt=""><figcaption></figcaption></figure> * Engineering standards turned in to automated checks which allowed visualizing service maturity * Management can track reliability and quality insights on their team's projects <figure><img src="/files/sUnmN6eqlYiGYa5Q9oKo" alt=""><figcaption></figcaption></figure> # Cross-team Collaboration 🤝 ## Stakeholder Engagement * **Product Engineering Teams** * API documentation requirements * Template feedback * Knowledge transfer and pair programming to deliver new templates * **Cloud Engineering** * Infrastructure templates (Terraform modules) * Security requirements (IAM, Service Accounts) * Resource provisioning (Atlantis/Terragrunt) * **Architecture Team** * Design patterns * Example: Modular templating pattern * Engineering standards * Example: Naming conventions, GitOps deployment pattern, Code Quality, Observability, Monitoring, etc. * **IT Department** * Authentication setup * Access management * Compliance requirements ## Communication Channels * Weekly sync meetings * Template review process * Feedback collection system # Lessons & Future 🎓 ## Key Learnings * Start with high-impact features * Friction is necessary when setting standards * Regular stakeholder alignment is needed ## Future Roadmap * Usage Insights <div align="left"><figure><img src="/files/xAdcgOmQDPMD60DhIDWo" alt="" width="375"><figcaption></figcaption></figure></div> * RBAC to control access to data and actions ## Challenges Overcome * Cross-team coordination and alignment * Competing priorities are hard to navigate * Communicating big changes to processes * Documentation, workshops, demos and more * Permission complexity * IAM and service accounts are tricky sometimes or maybe just in GCP * Template standardization * Different teams have unique needs so good customization options were necessary # Provisioning # Terraform Infrastructure as code (IaC) tools allow you to manage infrastructure with configuration files rather than through a graphical user interface. IaC allows you to build, change, and manage your infrastructure in a safe, consistent, and repeatable way by defining resource configurations that you can version, reuse, and share. Terraform is **HashiCorp's** infrastructure as code tool. It lets you define resources and infrastructure in human-readable, declarative configuration files, and manages your infrastructure's lifecycle. Using Terraform has several advantages over manually managing your infrastructure: * Terraform can manage infrastructure on multiple cloud platforms. * The human-readable configuration language helps you write infrastructure code quickly. * Terraform's state allows you to track resource changes throughout your deployments. * You can commit your configurations to version control to safely collaborate on infrastructure. Terraform's configuration language is declarative, meaning that it describes the desired end-state for your infrastructure, in contrast to procedural programming languages that require step-by-step instructions to perform tasks. Terraform providers automatically calculate dependencies between resources to create or destroy them in the correct order. # Installation [Homebrew](https://brew.sh/) is a free and open-source package management system for Mac OS X. Install the official [Terraform formula](https://github.com/hashicorp/homebrew-tap) from the terminal. First, install the HashiCorp tap, a repository of all our Homebrew packages ``` brew tap hashicorp/tap ``` Now, install Terraform with `hashicorp/tap/terraform`. ``` brew install hashicorp/tap/terraform ``` Install autocompletion ``` terraform -install-autocomplete ``` Update Terraform ``` brew upgrade terraform ``` # Usage ## [init](https://www.terraform.io/cli/commands/init) ``` terraform init ``` It’s  the  first  command  you  need  to  execute.  Otherwise,  terraform plan,  apply,  destroy  and  import  will  not  work. It is safe to run this command multiple times.  The  command will  : * find terraform  modules based on source provided * configure and validate  a  backend (if present) * install provider(s)  plugins ## [validate](https://www.terraform.io/cli/commands/validate) ``` terraform validate ``` Once you’ve initialized the directory, it’s a good idea to run the`validate` command before you run `plan` or `apply`. Validation catches syntax errors, version errors and more. > It is safe to run this command automatically, for example as a post-save check in a text editor or as a test step for a re-usable module in a CI system. ## [get](https://www.terraform.io/cli/commands/get) This  command  is  useful  when you have  some  modules defined. ```  terraform get  ``` The modules are downloaded into a `.terraform` subdirectory of the current working directory. **Don't commit this directory to your version control repository.** > `update=true` - If specified, modules that are already downloaded will be checked for updates and the updates will be downloaded if present. ## [plan](https://www.terraform.io/cli/commands/plan) The  plan  command creates an execution plan which allows previewing changes. Like a dry run, it won't actually apply any of the proposed changes. ```  terraform  plan  ``` > By default, when Terraform creates a plan it: > > * Reads the current state of any already-existing remote objects to make sure that the Terraform state is up-to-date. > * Compares the current configuration to the prior state and noting any differences. > * Proposes a set of change actions that should, if applied, make the remote objects match the configuration. ## [apply](https://www.terraform.io/cli/commands/apply) Next comes the apply command which will execute the actions proposed in the plan. ``` terraform apply ``` > `-auto-approve` - Skips interactive approval of plan before applying. This option is ignored when you pass a previously-saved plan file, because Terraform considers you passing the plan file as the approval and so will never prompt in that case. Apply only one resource ``` terraform  apply -target="module.s3" ``` ## [destroy](https://www.terraform.io/cli/commands/destroy) ```  terraform  destroy ``` This command deletes all the resources in the configuration. A  deletion  plan  can  be  created  before execution: ```  terraform  plan  –destroy ``` Deletions can take a target resource too. ``` terraform  destroy - target=" aws_s3_bucket.my_bucket" ``` # Configuration Management In DevOps, **configuration management** is a practice that involves **tracking** and **managing configurations** of infrastructure and software to ensure **consistency** and **reliability** across different environments. It relies on **automation**tools to streamline setup and updates, and **version control** systems to manage changes and enable rollbacks. Additionally, it supports **compliance** and **auditing** by monitoring configurations against organizational policies and providing an audit trail. # Ansible Ansible is an open-source configuration management, application deployment and provisioning tool that uses its own declarative language in YAML. Ansible is agentless, meaning you only need remote connections in order for it to function. [**Official Documentation**](https://docs.ansible.com/) # Benefits Ansible, a cutting-edge automation tool, proves invaluable for companies seeking efficient task and command automation across diverse nodes, including servers and PCs. Its versatility extends to both on-premises and cloud environments. ## Understanding Ansible in the Ecosystem * **Configuration Management vs. Provisioning:** While Ansible excels in configuration management, Terraform serves as a provisioning tool, creating and managing Cloud Infrastructure resources. * **TLDR:** Provision with Terraform. Configure with Ansible. ## Key Features * **Open Source and Community Support:** Ansible is freely available as an open-source tool (GitHub repository: [ansible/ansible](https://github.com/ansible/ansible)), boasting a robust and expansive community. * **Infrastructure as Code (IaC):** Commands, tasks, and codes seamlessly transition into Infrastructure as Code, enabling tasks to be saved, versioned, repeated, and tested. * **Declarative Configuration:** IaC adopts a 'Declarative Way,' allowing the definition of the desired configuration for streamlined management. * **Agentless Architecture:** Unlike traditional approaches, Ansible operates without requiring an agent app on the worker node. * **Parallel Execution:** Ansible inherently performs tasks in parallel across multiple hosts, optimizing efficiency. * **Cross-Platform Compatibility:** Supporting both Linux and Windows PCs, Ansible proves versatile in addressing various operating systems. * **Comprehensive Documentation:** Ansible offers well-designed documentation for users, facilitating a smoother learning and implementation process ([Ansible Documentation](https://docs.ansible.com/)). * **SSH Communication:** Ansible leverages SSH for seamless communication with other nodes. * **Module-Based Task Handling:** The tool efficiently manages tasks through a modular approach, enhancing its capability to handle diverse tasks. ## Integration with Other Technologies * **Integration with Terraform:** Ansible seamlessly integrates with various technologies, such as Terraform, expanding its capabilities and adaptability. In summary, Ansible stands out for its user-friendly and powerful automation capabilities, with seamless integration possibilities, making it a preferred choice for diverse tasks in both on-premises and cloud environments. # Installation {% embed url="<https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-and-upgrading-ansible-with-pipx>" %} Pipx installs and runs Python applications in isolated environments. ```bash brew install pipx pipx ensurepath pipx install --include-deps ansible # full version pipx install ansible-core # minimal version ``` At this point you should be able to get the version of ansible in a new shell session. ```bash ansible --version # check if it worked ``` # Build Systems # Bazel Intro Bazel is an **opinionated** **build system** that I’ve come to appreciate for its efficiency and scalability. Originally **developed by Google** to tackle the challenge of managing extensive codebases across various languages and platforms, Bazel is now available as **open-source**. Official documentation can be found here -> <https://docs.bazel.build> In the subsections, I will cover concepts I've learned and provide tips. # Features <details> <summary>Explicit Dependency Management</summary> #### What is Explicit Dependency Management? In Bazel, **Explicit Dependency Management** means that you must clearly and explicitly list all the dependencies that your build targets need. Unlike some other build systems where dependencies might be inferred automatically, Bazel requires you to specify each dependency directly. The dependency graph in Bazel must always be a **directed acyclic graph** which means no cycles are allowed. #### Basic Idea * **Clear Declaration:** You must list every file, library, or target your build depends on in your build configuration files. This ensures that Bazel knows exactly what each target needs to build correctly. * **No Implicit Dependencies:** There is no automatic assumption about what dependencies a target might need based on file locations or other heuristics. Everything must be explicitly stated. #### Example Scenario Let’s say you’re working on a project with a library and an application that depends on that library. Here’s how you’d manage dependencies in Bazel: 1. **Project Structure:** ```less lessCopy codemy_project/ ├── //app │ ├── BUILD.bazel │ └── main.py └── //lib ├── BUILD.bazel └── utils.py ``` 2. **Targets and Dependencies:** * **`//lib:utils`** - A target that builds a library with utility functions. * **`//app:main`** - A target that builds an application and depends on the utility functions. 3. **Setting Up Explicit Dependencies:** **`//lib/BUILD.bazel`** ```python py_library( name = "utils", srcs = ["utils.py"], ) ``` **`//app/BUILD.bazel`** ```python py_binary( name = "main", srcs = ["main.py"], deps = ["//lib:utils"], # Explicitly declare the dependency on //lib:utils ) ``` #### Explanation * **Explicit Declaration:** * In the `//lib/BUILD.bazel` file, the `py_library` target `//lib:utils` is defined with its source files. This target represents the library with utility functions. * In the `//app/BUILD.bazel` file, the `py_binary` target `//app:main` is defined as a Python binary. The `deps`attribute explicitly lists `//lib:utils` as a dependency. This means that `//app:main` requires `//lib:utils` to build. * **No Implicit Dependencies:** * Bazel doesn’t guess or infer that `//app:main` might need `//lib:utils` based on file paths or other heuristics. You must explicitly state this dependency in the `deps` attribute. #### Benefits * **Predictability:** By explicitly declaring all dependencies, you ensure that your build is predictable and reproducible. You know exactly what each target depends on. * **Clarity:** It makes the build configuration clear and understandable. Anyone looking at the build files can see all the dependencies listed without guessing. * **Error Prevention:** It reduces the risk of build errors due to missing dependencies. Since everything is declared explicitly, Bazel will catch any missing dependencies and prompt you to fix them. #### Comparison to Implicit Dependencies * **Implicit Dependencies:** In some build systems, dependencies might be inferred based on file locations or naming conventions. This can lead to unexpected issues if the system makes incorrect assumptions. * **Explicit Dependencies (Bazel):** Bazel’s requirement for explicit declarations avoids such issues. You specify exactly what each target needs, which makes the build process more reliable. #### Summary **Explicit Dependency Management** in Bazel means you must clearly list every dependency your build targets require. This approach ensures that your build process is predictable, clear, and free from implicit assumptions about what dependencies are needed. By specifying dependencies directly in your build configuration files, you avoid potential issues and make your project easier to manage. </details> <details> <summary>Advanced Visibility Features</summary> #### What is Visibility in Bazel? In Bazel, visibility is a feature that allows you to control which parts of your code can access or depend on specific build targets. This helps you manage and restrict how different parts of your project interact with each other, improving modularity and reducing the risk of unintended dependencies. #### Basic Idea * **Visibility Control:** You can limit which packages or targets are allowed to depend on a particular target. This way, you can keep certain parts of your codebase hidden from others and only expose what's necessary. #### Example Scenario Imagine you have a project with multiple modules: 1. **Project Structure:** ```less my_project/ ├── //frontend │ ├── BUILD.bazel │ └── main.py ├── //backend │ ├── BUILD.bazel │ └── service.py └── //shared ├── BUILD.bazel └── utils.py ``` 2. **Targets and Visibility:** * **`//frontend:main`** - A target that builds the frontend module. * **`//backend:service`** - A target that builds the backend module. * **`//shared:utils`** - A target that builds some shared utility functions. 3. **Setting Up Visibility:** * You want to allow only the `frontend` module to use the utilities from `shared`, but you don’t want the `backend` module to have access to these utilities. Here’s how you can set it up in Bazel: **`//shared/BUILD.bazel`** ```python pythonCopy codepy_library( name = "utils", srcs = ["utils.py"], visibility = ["//frontend:__pkg__"], # Only the frontend module can use this target ) ``` **`//frontend/BUILD.bazel`** ```python pythonCopy codepy_binary( name = "main", srcs = ["main.py"], deps = ["//shared:utils"], # Can depend on utils ) ``` **`//backend/BUILD.bazel`** ```python pythonCopy codepy_binary( name = "service", srcs = ["service.py"], # No dependency on //shared:utils, as it's not listed here ) ``` #### Explanation * **Visibility Declaration:** * In the `//shared/BUILD.bazel` file, the `visibility` attribute for `//shared:utils` is set to `["//frontend:__pkg__"]`. This means that only targets in the `frontend` package can see and depend on the `//shared:utils` target. * **Controlled Access:** * The `frontend` module can use the `utils` library because it is explicitly allowed by the visibility setting. * The `backend` module cannot access the `utils` library because it’s not listed in the visibility setting. This prevents unintended dependencies and keeps the `backend` module isolated from the `shared` utilities. #### Benefits * **Modularity:** By controlling visibility, you can better organize your project and enforce modular design. Each part of the project only has access to the dependencies it needs. * **Maintainability:** It reduces the risk of changes in one part of the project affecting others unintentionally, making the project easier to maintain and refactor. #### Summary Bazel’s visibility feature lets you control which parts of your project can access specific build targets. By using visibility, you can limit dependencies and keep different parts of your codebase modular and isolated, similar to how you might use package visibility in Java or namespaces in C++. </details> **TODO:** * Remote Build Execution and Caching * Build Dependency Analysis * Fast, Correct Builds (and Tests) # Secure Software Engineering As a software engineer, I know how important it is to build secure software in today's world. The CSSLP training I've completed has given me the skills to create software that's safe from cyber threats.\ \ The content presented in the following sections is based on my learning from the "All-in-One is All You Need" CSSLP exam guide. \ \ This comprehensive guide has been an invaluable resource in my journey to understand and master the concepts of secure software development and secure software lifecycle management. It provides comprehensive insights into the CSSLP domains and serves as a foundation for my knowledge in this critical field. # Core Concepts The Secure Software Concepts domain encompasses essential principles from information security that are crucial for secure software development, including **confidentiality**, **integrity and** **availability** (CIA). To fully describe operational security elements, authentication, authorization, auditability and nonrepudiation have been incorporated into the existing CIA framework. * **Confidentiality**: Ensuring that information is only accessible to those with the proper authorization. * **Integrity**: Maintaining the accuracy and reliability of data and systems, preventing unauthorized tampering or alterations. * **Availability**: Ensuring that systems and data are consistently accessible and operational when needed. * **Authentication**: Verifying the identity of users or entities accessing a system or data. * **Authorization**: Granting or denying specific permissions and access rights to users or entities based on their authenticated identity. * **Auditability**: The capability to track and record actions and events within a system for monitoring and accountability purposes. * **Nonrepudiation**: Preventing individuals from denying their actions or transactions, ensuring accountability and trustworthiness. ### Secure Development Lifecycle Adding team awareness and education, gates and security requirements, bug tracking, threat modeling, fuzzing, security reviews and mitigations to the software development lifecycle makes it more secure. #### Team Awareness & Education All team members should undergo ongoing role-specific training in both basic security knowledge, essential for the entire team, and advanced topics tailored to their specific roles to ensure they remain up-to-date and equipped to address evolving security challenges throughout their careers. #### Gates & Security Requirements During the development process, periodic security reviews, known as gates, are conducted to ensure adherence to security requirements. #### Bug Tracking As code is being developed, it's common to uncover bugs, which are code components exhibiting problematic behaviors, occasionally presenting exploitable vulnerabilities, and necessitating tracking and resolution. #### Threat Modeling Threat modeling serves to comprehensively identify and describe system threats while recording mitigation strategies. #### Fuzzing Fuzzing is an automated testing technique involving the application of diverse inputs to an interface, examining the outputs for unexpected behaviors, commonly used by both security testers and hackers to discover vulnerabilities and potential exploits. #### Security Reviews Security reviews serve as checkpoints within the process to ensure these steps are not bypassed and are functioning correctly. #### Mitigations Adopting a risk assessment model like DREAD, which calculates risk as a product of impact and probability, aids in prioritizing bug fixes, with the mitigation process offering four standard options for addressing security bugs: **do nothing**, **warn the user**, **remove the problem**, and **fix the problem**. #### Software Security vs Quality High-quality software can be defect-free but may not necessarily be secure. Conversely, the critical concern is that if software lacks quality and contains defects, it is unlikely to be secure. <figure><img src="/files/Qkj4qMbcHV27caMzuzyL" alt=""><figcaption>Security vs Quality</figcaption></figure> # Security Design Principles Secure designs are built upon a solid groundwork of secure design principles: * **Good enough security** * There is an appropriate level of required security for systems * **Least privilege** * Users and applications should have only the necessary rights and privileges to perform their current tasks. * **Separation of duties** * Software components should require multiple conditions to pass before a task can be considered as complete. This ensures no single individual can abuse the system * **Defense in depth** * Defense in depth is a security principle involving the use of multiple overlapping layers of protection to enhance security, with the understanding that no single defense mechanism is foolproof, and the goal is to make compromising a system more costly and time-consuming for adversaries. * **Fail-safe** * Fail-safe, in the context of system design, means that when a system encounters a failure, it should transition to a secure and stable state. * **Economy of mechanism** * Systems and security mechanisms should be kept as simple as possible because complexity increases the potential for vulnerabilities and makes troubleshooting more challenging. * **Complete mediation** * Complete mediation is the security principle that requires continuous verification of a subject's authorization for each access request to an object and action. * **Open design** * Open design is the principle that a system's security should not rely on the secrecy of its design or algorithms but instead on elements like keys, making the security independent of the design's obscurity. * **Least common mechanism** * Least common mechanism is a design approach aimed at preventing unintentional information sharing among processes by minimizing shared components, favoring separate processes for distinct functions to enhance security. * **Psychological acceptability** * A security system should not burden users as it will cause them to work around security aspects. * **Weakest link** * Common point of failure of a system. A system is only as strong as the weakest link. * **Leverage existing components** * New components increase the chances of new vulnerabilities being added. Therefore, reusing components should decrease the chances. * **Single point of failure** * Software systems should not have a single point of failure. # Software Security Requirements Requirements are like the blueprint of software, setting expectations for how it should function and operate. ### Functional Requirements Functional Requirements in software development specify what the software should do or the functions it should perform. Here are a few standard functional requirements the DevOps team would have: * Deployment platform requirements * Database requirements * Disaster recovery/business continuity planning (DR/BCP) requirements * Infrastructure requirements **Example:** "The system must allow users to log in using a username and password, and upon successful login, they should have access to their personal account information." ### Non-Functional Requirements Non-Functional Requirements, on the other hand, define how the system should perform its functions. They describe the quality attributes of the system, such as performance, security, scalability, and usability. In the context of security, a non-functional requirement could be: **Example:** "The system must encrypt all sensitive user data, such as passwords and financial information, using industry-standard encryption algorithms (e.g., AES-256) to ensure data confidentiality." ### Operational and Deployment Requirements Operational and deployment requirements in software development pertain to the considerations of how software will operate and be deployed within an enterprise environment, where it often coexists with established technology standards, such as specific platforms, operating systems, and infrastructure components. ### Use Cases Use cases are a valuable method for expressing functional requirements in a way that developers and testers can easily understand. They represent specific instances of how the system should behave and are particularly useful for describing complex or unclear interactions with the system, ensuring that both software design and testing adequately address potential issues arising from poorly defined requirements. Example: In an e-commerce application, a use case might be "User places an order," which describes the step-by-step process of how a user selects products, adds them to the cart, provides shipping information, and completes the purchase. This use case helps clarify the functional requirements for this specific user interaction. # Compliance Standards and Policies Regulations and compliance refer to the rules and standards that govern various aspects of an enterprise's operations, including those related to software development. Compliance with these rules is essential because failure to adhere to them can result in financial penalties, increased scrutiny, future regulatory constraints, and negative publicity. Compliance is distinct from security, as it focuses on adhering to established rules, even though strict compliance does not necessarily guarantee security. ## Security standards organizations ### ISO ISO stands for the International Organization for Standardization. It is an international organization that develops and publishes standards to ensure the quality, safety, efficiency, and interoperability of products, services, and systems across various industries, including information security. ### IEC The International Electrotechnical Commission (IEC) is an international standards organization that specializes in developing and publishing standards related to electrical and electronic technologies, components, and systems. #### ISO/IEC 9126 Quality Characteristics This ISO/IEC standard defines six quality characteristics that can be used to measure the quality of software: * Functionality * Reliability * Usability * Efficiency * Maintainability * Portability ### SAFECode SAFECode is an industry-supported organization that promotes collaboration among companies to enhance software assurance practices by sharing successful best practices and providing guidance on secure software development. [**Practical Security Stories**](http://safecode.org/wp-content/uploads/2018/01/SAFECode_Agile_Dev_Security0712.pdf) ### **OWASP** OWASP is a global, open community dedicated to enhancing application software security by publishing Top Ten vulnerability lists and providing valuable resources on their website ([www.owasp.org](http://www.owasp.org/)) to help organizations create more secure software. ### NIST NIST, or the National Institute of Standards and Technology, is a federal agency responsible for developing technology, measurements, and standards **aligned with the U.S. economy's interests**. \ \ Within NIST, the Computer Security Division addresses computer security concerns, including compliance with laws like the Federal Information Security Management Act (FISMA). NIST publishes key document types like Federal Information Processing Standards (FIPS) and the Special Publication (SP) 800 series, providing guidelines for information security practices, cryptographic protocols, and risk management frameworks, making them valuable resources for the information systems community Some important publications are: * FIPS 200 * FIPS 199 * FIPS 197 * FIPS 186-3 * FIPS 190-4 * FIPS 140 Series * SP 800-152 * SP 800-107 * SP 800-100 * SP 800-63 * SP 800-53 * SP 800-30 * SP 800-12 * SP 800-218 (SSDF) # Sarbanes-Oxley (SOX) In summary, a security software engineer should know that SOX, particularly Section 404, places significant emphasis on the security of financial reporting systems. They should understand the requirements, control measures, documentation, testing, and consequences of non-compliance related to information security under SOX to effectively contribute to their organization's compliance efforts. ## **Background** SOX, officially known as the Sarbanes-Oxley Act of 2002, was enacted in response to corporate scandals like Enron and WorldCom, which undermined investor confidence. It is a U.S. federal law that sets requirements for financial reporting and corporate governance. ## **Section 404** This is a crucial section of SOX for security professionals. It requires companies to establish and maintain adequate internal controls over financial reporting systems. In other words, it mandates that companies have security measures in place to ensure the accuracy and integrity of financial data. Security software engineers should understand the importance of data integrity in financial reporting. ## **Control Measures** Engineers should be familiar with the types of control measures that can help achieve compliance with Section 404. This includes implementing access controls, encryption, audit trails, and other security mechanisms to protect financial data from unauthorized access, tampering, or fraud. ## **Documentation** SOX compliance involves thorough documentation of security policies, procedures, and controls. Engineers should know how to create and maintain clear and comprehensive documentation to demonstrate compliance. ## **Testing and Auditing** Section 404 requires regular testing and auditing of internal controls to ensure they are effective. Security engineers should be prepared to assist in these efforts, helping to identify weaknesses and vulnerabilities in security controls. ## **Penalties for Non-Compliance** Engineers should understand the potential consequences of non-compliance with SOX. Failure to meet the requirements can result in financial penalties, legal liabilities, and damage to a company's reputation. ## **Impact on IT Systems** Engineers need to consider how SOX compliance affects IT systems, especially those involved in financial reporting. They should ensure that security measures are integrated into these systems to meet SOX requirements. ## **Ongoing Compliance** SOX compliance is an ongoing process. Engineers should be aware that maintaining compliance requires continuous monitoring, assessment, and improvement of security controls. # HIPAA and HITECH In summary, security software engineers should be well-versed in the HIPAA Security Rule, the breach notification requirements of the HITECH Act, and the broader security best practices for protecting electronic personal health information (ePHI). This knowledge is critical for designing, developing, and maintaining healthcare-related software systems that handle sensitive patient information securely and in compliance with these important healthcare regulations. ## **Purpose** HIPAA and HITECH are U.S. federal laws that regulate the privacy and security of protected health information (PHI) and promote the adoption of electronic health records (EHRs). ### **HIPAA** #### **Security Rule** Engineers should be familiar with the HIPAA Security Rule, which sets standards for safeguarding electronic PHI (ePHI). This includes requirements for access controls, encryption, audit logging, and other security measures. #### **Risk Assessment** Engineers should understand the importance of conducting regular risk assessments to identify and mitigate security risks to ePHI. This is a fundamental requirement under HIPAA. #### **Business Associate Agreements** HIPAA mandates that covered entities (e.g., healthcare providers) have business associate agreements in place with third parties (e.g., software vendors) who handle ePHI. Engineers should know the security obligations these agreements entail. ### **HITECH Act** #### **Breach Notification** Engineers should be aware of the HITECH Act's breach notification requirements, which mandate that covered entities report breaches of unsecured PHI to affected individuals, the Department of Health and Human Services (HHS), and, in some cases, the media. #### **Increased Penalties** HITECH introduced higher penalties for non-compliance with HIPAA, making it essential for engineers to help ensure security measures are in place to avoid these penalties. ### **Security Best Practices** * **Access Control:** Engineers should understand the importance of implementing robust access controls to limit access to ePHI based on the principle of least privilege. * **Data Encryption:** They should know the significance of encrypting ePHI both in transit and at rest to protect it from unauthorized access. * **Audit Logging:** Engineers should be familiar with audit logging requirements to track and monitor access to ePHI, helping in identifying and responding to security incidents. * **Incident Response:** Engineers should be prepared to assist in developing and implementing incident response plans to address security breaches or incidents involving ePHI. * **Training and Awareness:** HIPAA and HITECH emphasize the need for workforce training and awareness programs to educate employees about security policies and procedures. Engineers can contribute to these efforts. * **Ongoing Compliance:** Compliance with HIPAA and HITECH is an ongoing process. Engineers should understand the need for continuous monitoring, periodic risk assessments, and updates to security measures to maintain compliance. # Payment Card Industry Data Security Standard (PCI-DSS) In summary, a security software engineer should be well-versed in the PCI DSS standards, including the specific security requirements, best practices for secure software development, incident response, and the compliance and validation processes. This knowledge is vital for designing, developing, and maintaining software systems that handle payment card data securely and in compliance with PCI DSS standards. ## **PCI DSS Overview** **Purpose:** Payment Card Industry (PCI) Data Security Standard (DSS) is a set of security standards established to protect payment card data, including credit card and debit card information, during processing, transmission, and storage. **Scope:** Engineers should recognize that PCI DSS applies to any organization that accepts, processes, stores, or transmits payment card data. This includes retailers, e-commerce sites, payment processors, and financial institutions. ## **Key Security Requirements** **Data Encryption:** Engineers should understand the importance of encrypting cardholder data both in transit and at rest. They should be familiar with encryption standards and protocols. **Access Control:** Knowledge of access control mechanisms, such as role-based access and least privilege principles, is essential for ensuring that only authorized individuals can access payment card data. **Network Security:** Engineers should be aware of network segmentation and firewall requirements to protect cardholder data environments (CDE) from unauthorized access. **Vulnerability Management:** Understanding vulnerability assessment and patch management is crucial for addressing security vulnerabilities promptly. ## **Secure Software Development** **Secure Coding Practices:** Engineers should follow secure coding practices to develop applications that handle payment card data securely. This includes input validation, secure authentication, and secure error handling. **Change Management:** Knowledge of change control processes and the impact of code changes on security is important to maintain PCI DSS compliance. ## **Incident Response** **Incident Handling:** Engineers should be prepared to assist in developing and implementing an incident response plan to address security incidents involving payment card data. ## **Logging and Monitoring** **Audit Trails:** Engineers should understand the importance of audit trails and logging to monitor and detect suspicious activities related to payment card data. **Security Information and Event Management (SIEM):** Familiarity with SIEM tools and practices is crucial for real-time monitoring and alerting. ## **Compliance and Validation** **Self-Assessment Questionnaire (SAQ):** Engineers should know about SAQs, which are used to assess compliance for organizations that do not require a full-scale audit. **Annual Assessments:** Engineers should be aware that organizations must undergo annual PCI DSS assessments, either through self-assessment or a Qualified Security Assessor (QSA) for larger entities. ## **Penalties and Liabilities** **Non-Compliance:** Engineers should understand the consequences of non-compliance with PCI DSS, which can include fines, restrictions, or termination of the ability to process payment card transactions. # General Data Protection Regulation (GDPR) Compliance with GDPR requires a holistic approach to data protection and security. Security software engineers should collaborate with legal and compliance experts to ensure that software systems align with GDPR requirements, as non-compliance can result in significant fines and reputational damage. ## **Scope of GDPR** Understand that GDPR applies to organizations, including software providers, that process personal data of individuals within the EU, regardless of where the organization is based. Personal data is defined as any information relating to an identified or identifiable natural person. This includes: * Online identifiers * IP addresses * Cookies It also includes indirect information, including physical, physiological, genetic, mental, economic, cultural, or social identities that can be traced to one person. ## Obligations GDPR mandates that individuals have the right to access clear and comprehensible information regarding the processing of their data. When organizations collect data from individuals, they are obligated to transparently convey key details, including: * The identity and contact information of the requesting entity * The purpose and usage of the data * The retention period * Potential international transfers * **The individual's rights such as access, rectification, erasure (right to be forgotten), withdrawal of consent, and the ability to file complaints.** # California Consumer Privacy Act (CCPA) This is California's bill that is similar to the GDPR. Under the CCPA, consumers should have the following rights: * Possess the right to understand the purpose of personal data usage. * Possess the right to be informed about and object to data sales. * Possess the right to know the recipients of their data. * Be free from discrimination for objecting to data sales. * Possess the right to access their held data. # Federal Risk and Authorization Management Program (FedRAMP) FedRAMP, short for the Federal Risk and Authorization Management Program, is a U.S. government program established to standardize the security assessment, authorization, and continuous monitoring processes for cloud products and services. Its primary goal is to ensure that cloud solutions used by federal agencies meet consistent and stringent security standards. Based on **NIST Special Publication 800-53**, it defines security controls and families. Controls address aspects like Access Control, Communications Protection, and Security Assessment. ### FedRAMP Marketplace The FedRAMP Marketplace is an online repository that provides a comprehensive list of cloud products and services that have received authorization through the Federal Risk and Authorization Management Program (FedRAMP). It serves as a valuable resource for federal agencies and organizations seeking FedRAMP-compliant cloud solutions. Examples of FedRAMP Authorized Cloud Services: * **Amazon Web Services (AWS)** * **Microsoft Azure** * **Google Cloud Platform (GCP)** * **Salesforce Government Cloud** * **ServiceNow** * **Adobe Document Cloud for Government** * **Oracle Cloud Infrastructure (OCI)** ### # Privacy & Data Privacy refers to an individual's right to keep their personal information, activities, and communications confidential. ## Personally Identifiable Information (PII) This refers to information that can be used to identify someone. * Full name * National identification number (i.e., SSN) * IP address * Home address * Motor vehicle registration plate number * Driver’s license or state ID number * Face, fingerprints, or handwriting * Credit card and bank account numbers * Date of birth * Birthplace * Genetic information ## Data Data refers to raw and unprocessed facts, figures, symbols, or information that can be in the form of numbers, text, images, or any other format. Data can be collected, stored, and manipulated to extract meaningful insights, support decision-making, or perform various tasks. ### Data Classification Data classification is a multifaceted process, encompassing state, use, and security importance, serving as a risk management tool to align protection costs with asset value and potentially becoming more intricate for larger enterprises with diverse data protection requirements, including compliance considerations. #### Data States * At Rest/Being Stored * Being Created/Generated * Being Transmitted * Being Updated/Deleted #### Data Usage * Input * Output * Initialized (internal) #### Data Risk Impact | Impact | Damage | Financial Consequence | | ------ | ------------------------------------- | ----------------------- | | High | Death | Greater than $1,000,000 | | Medium | Severe Injury / Loss of Functionality | Greater than $100,000 | | Low | Minor Injury | Less than $100,000 | # Introduction to Linux Here are some core concepts behind Linux that are important to know about: ## Concepts * Kernel * Distribution * Boot Loader * Service * Filesystem * X Window System * Desktop Environment * Command Line ## Kernel **Think of it as the brain of your computer.** It manages hardware, system resources, and communication between software and hardware. <div align="left" data-full-width="false"><figure><img src="/files/s741YJ9ES6vIDuhD8NCm" alt="" width="188"><figcaption></figcaption></figure></div> ## Distribution A version of Linux that includes **the linux kernel plus additional software, tools, and package management.** It's like a customized Linux operating system, such as Ubuntu or Fedora. ## Boot Loader A small program that runs first when you start your computer, loading the operating system (like GRUB). It's the gatekeeper to the OS. ## Service A program or process that runs in the background to provide functionality (e.g., web server, database). Services keep the system running smoothly without direct interaction. ## Filsystem The way data is stored and organized on your computer. It's like the file cabinet of your computer where everything is saved and accessed. ## X Window System The system that handles the graphical display on your screen (windows, icons, etc.) and enables interaction with input devices (like the mouse and keyboard). It's **the foundation** for your graphical user interface (GUI), but **doesn't include the look and feel of the interface itself**. ## Desktop Environment The visual user interface that lets you interact with your computer (e.g., GNOME, KDE). It includes icons, windows, and apps, making your system **visually functional**. ## Command Line A text-based interface to interact with the system by typing commands. It's a powerful, faster way to control your computer compared to clicking around. # Architecture <img src="/files/NZdNI3qwEgu8ZASgaHq2" alt="" data-size="original"> * The Linux kernel is monolithic in nature. * System calls are used to interact with the Linux kernel space. * Kernel code can only be executed in the kernel mode. Non-kernel code is executed in the user mode. * Device drivers are used to communicate with the hardware devices. # Server Administration ### <a href="#multi-user-operating-systems" id="multi-user-operating-systems"></a> # User / Groups ### Multi-User Operating Systems <a href="#multi-user-operating-systems" id="multi-user-operating-systems"></a> An operating system is considered as multi-user if it allows multiple people/users to use a computer and not affect each other's files and preferences. Linux based operating systems are multi-user in nature. ### User/Group Management <a href="#usergroup-management" id="usergroup-management"></a> * Users in Linux has an associated user ID called UID attached to them. * A group is a collection of one or more users. * A group makes it easier to share permissions among a group of users. * Each group has a group ID called GID associated with it. #### id command <a href="#id-command" id="id-command"></a> `id` command can be used to find the uid and gid associated with an user. It also lists down the groups to which the user belongs to. The uid and gid associated with the root user is 0. ![](/files/tI0F78FG7bvp4sUOppSi) A good way to find out the current user in Linux is to use the whoami command. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image35.png) **"root" user or superuser is the most privileged user with** **unrestricted access to all the resources on the system. It has UID 0** Important files associated with users/groups | /etc/passwd | Stores the user name, the uid, the gid, the home directory, the login shell etc | | ----------- | ---------------------------------------------------------------------------------- | | /etc/shadow | Stores the password associated with the users. Can only be accessed by super users | | /etc/group | Stores information about different groups on the system | ### Important commands for managing users <a href="#important-commands-for-managing-users" id="important-commands-for-managing-users"></a> Some of the commands which are used frequently to manage users/groups on Linux are following: * `useradd` - Creates a new user * `passwd` - Adds or modifies passwords for a user * `usermod` - Modifies attributes of a user (like home directory or shell) * One easy way of providing root access to users is to add them to a group which has permissions to run all the commands. "wheel" is a group in redhat Linux with such privileges. `usermod -a -G wheel <user>` * `userdel` - Deletes a user * `su` - Switch user ### Important commands for managing groups <a href="#important-commands-for-managing-groups" id="important-commands-for-managing-groups"></a> | groupadd \\\<group\_name> | Creates a new group | | ------------------------- | ------------------------------ | | groupmod \\\<group\_name> | Modifies attributes of a group | | groupdel \\\<group\_name> | Deletes a group | | gpasswd \\\<group\_name> | Modifies password for group | # File Permissions On a Linux operating system, each file and directory is assigned access permissions for the owner of the file, the members of a group of related users and everybody else. This is to make sure that one user is not allowed to access the files and resources of another user. ![](/files/6X889or3XOfF5cfoPkj4) ![](/files/d2XVPPX7K6w4CNXeVEpX) ![](/files/QK22VFL8BcJ9Xaw6Z0ow) #### Chmod command <a href="#chmod-command" id="chmod-command"></a> The chmod command is used to modify files and directories permissions in Linux. The chmod command accepts permissions in as a numerical argument. We can think of permission as a series of bits with 1 representing True or allowed and 0 representing False or not allowed. | Permission | rwx | Binary | Decimal | | ----------------------- | --- | ------ | ------- | | Read, write and execute | rwx | 111 | 7 | | Read and write | rw- | 110 | 6 | | Read and execute | r-x | 101 | 5 | | Read only | r-- | 100 | 4 | | Write and execute | -wx | 011 | 3 | | Write only | -w- | 010 | 2 | | Execute only | --x | 001 | 1 | | None | --- | 000 | 0 | Each digit is independent of the other two. Therefore, `750` means the current user can read, write, and execute, the group cannot write, and others cannot read, write, or execute. `744`, which is a typical default permission, allows read, write, and execute permissions for the owner, and read permissions for the group and “world” users. #### `664 => rw-rw-r--` <a href="#chmod-664-rw-rw-r" id="chmod-664-rw-rw-r"></a> ``` chmod 664 example.txt ``` \ **Chown command** The chown command is used to change the owner of files or directories in Linux. Command syntax: chown \\\<new\_owner> \\\<file\_name> ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image6.png) **Chgrp command** The chgrp command can be used to change the group ownership of files or directories in Linux. The syntax is very similar to that of chown command. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image27.png) Chgrp command can also be used to change the owner of a directory in the similar way.\ # SSH The ssh command is used for logging into the remote systems, transfer files between systems and for executing commands on a remote machine. SSH stands for secure shell and is used to provide an encrypted secured connection between two hosts over an insecure network like the internet. Reference: <https://www.ssh.com/ssh/command/> #### Passwordless Authentication Using SSH <a href="#passwordless-authentication-using-ssh" id="passwordless-authentication-using-ssh"></a> Using this method, we can ssh into hosts without entering the password. This method is also useful when we want some scripts to perform ssh-related tasks. Steps for setting up a passwordless authentication with a remote host: 1. Install openssh package which contains all the commands related to ssh. 2. Generate a key pair using the ssh-keygen command. 3. After running the ssh-keygen command successfully, we should see two keys present in the `~/.ssh` directory. `id_rsa` is the private key and `id_rsa.pub` is the public key. Do note that the private key can only be read and modified by you. 4. Transfer the public key to the remote host 1. There are multiple ways to transfer the public key to the remote server. We will look at one of the most common ways of doing it using the ssh-copy-id command. 2. Install the openssh-clients package to use ssh-copy-id command. 5. Our public key should be in `~/.ssh/authorized_keys` now. `~/.ssh/authorized_key` contains a list of public keys. The users associated with these public keys have the ssh access into the remote host. #### How to run commands on a remote host ? <a href="#how-to-run-commands-on-a-remote-host" id="how-to-run-commands-on-a-remote-host"></a> General syntax: ssh \\\<user>@\\\<hostname/hostip> \\\<command> #### How to transfer files from one host to another host ? <a href="#how-to-transfer-files-from-one-host-to-another-host" id="how-to-transfer-files-from-one-host-to-another-host"></a> General syntax: scp \\\<source> \\\<destination> # Process Management Some useful commands that can be used to monitor the processes on Linux systems. #### ps (process status) <a href="#ps-process-status" id="ps-process-status"></a> The ps command is used to know the information of a process or list of processes. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image24.png) If you get an error "ps command not found" while running ps command, do install **procps** package. ps without any arguments is not very useful. Let's try to list all the processes on the system by using the below command. Reference: <https://unix.stackexchange.com/questions/106847/what-does-aux-mean-in-ps-aux> ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image42.png) We can use an additional argument with ps command to list the information about the process with a specific process ID. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image2.png) We can use grep in combination with ps command to list only specific processes. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image1.png) **top** The top command is used to show information about Linux processes running on the system in real time. It also shows a summary of the system information. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image53.png) For each process, top lists down the process ID, owner, priority, state, cpu utilization, memory utilization and much more information. It also lists down the memory utilization and cpu utilization of the system as a whole along with system uptime and cpu load average.\ \ **Memory Management** In this section, we will study about some useful commands that can be used to view information about the system memory. #### free <a href="#free" id="free"></a> The free command is used to display the memory usage of the system. The command displays the total free and used space available in the RAM along with space occupied by the caches/buffers. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image22.png) free command by default shows the memory usage in kilobytes. We can use an additional argument to get the data in human-readable format. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image5.png) Checking Disk Space In this section, we will study about some useful commands that can be used to view disk space on Linux. #### df (disk free) <a href="#df-disk-free" id="df-disk-free"></a> The df command is used to display the free and available space for each mounted file system. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image36.png) #### du (disk usage) <a href="#du-disk-usage" id="du-disk-usage"></a> The du command is used to display disk usage of files and directories on the system. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image10.png) The below command can be used to display the top 5 largest directories in the root directory. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image18.png) Systemd Systemd is a system and service manager for Linux operating systems. Systemd units are the building blocks of systemd. These units are represented by unit configuration files. The below examples shows the unit configuration files available at /usr/lib/systemd/system which are distributed by installed RPM packages. We are more interested in the configuration file that ends with service as these are service units. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image16.png) #### Managing System Services <a href="#managing-system-services" id="managing-system-services"></a> Service units end with .service file extension. Systemctl command can be used to start/stop/restart the services managed by systemd. | Command | Description | | ------------------------------ | ------------------------------------- | | systemctl start name.service | Starts a service | | systemctl stop name.service | Stops a service | | systemctl restart name.service | Restarts a service | | systemctl status name.service | Check the status of a service | | systemctl reload name.service | Reload the configuration of a service | ### Logs <a href="#logs" id="logs"></a> In this section, we will talk about some important files and directories which can be very useful for viewing system logs and applications logs in Linux. These logs can be very useful when you are troubleshooting on the system. ![](https://linkedin.github.io/school-of-sre/level101/linux_basics/images/linux/admin/image58.png) \ \ \ # Networking # Diagrams ![](/files/oQbfv4lajOudQZ7fFe96) <img src="/files/HJeZu28CE8U7jHTmmvyX" alt="" data-size="original"> {% embed url="<https://media.licdn.com/dms/image/v2/D4D22AQHpYiUURLgI9g/feedshare-shrink_800/feedshare-shrink_800/0/1703939855448?e=1730332800&t=OVKffRHldUStGpGG61a-FDtse2wPXZ2VTTRWGyf99kM&v=beta>" %} # Browser URL Example What happens when you start typing google in your browser's search bar? <https://github.com/alex/what-happens-when> ### Simplified #### Keyboard Actions * **Key Pressed**: "g" key pressed; browser processes it and suggests URLs based on history. * **Enter Key**: Physical keypress leads to electrical signals transmitted to the computer. * USB keyboard: Current flows to logic circuitry, keycode is generated and sent. * Virtual keyboard: Capacitive touch registers keypress, sends a software interrupt. #### OS-Level Events * **Interrupts**: * OS detects the keypress and triggers an interrupt request. * **Windows**: Sends a `WM_KEYDOWN` message. * **macOS**: Sends an `NSEvent`. * **Linux**: Xorg server listens for keycodes. #### URL Parsing * **URL or Search Term**: Determines if the input is a URL or search term. * **Non-ASCII Characters**: Converts characters using Punycode if needed. #### Network Communication * **DNS Lookup**: * If the browser doesn't find the domain in its local cache, it sends a request to the DNS server (usually provided by the ISP or a local router). * If the DNS server doesn't have the answer, it will act as a recursive resolver, querying other DNS servers higher up in the hierarchy on behalf of the client. <figure><img src="/files/CFIyknl7GSIjhhRyJteM" alt=""><figcaption></figcaption></figure> * **ARP Process**: Determines the MAC address for the DNS server or default gateway. * **Socket Opening**: Establishes a connection to the server using TCP. * **TLS Handshake**: Encrypts communication using SSL/TLS for HTTPS. #### HTTP Protocol * **Request**: Sends an HTTP request to the server (e.g., GET request). * **Response**: Server returns HTML, possibly with a status code (e.g., `200 OK`). * **Resources**: Browser fetches additional resources (images, CSS) as needed. #### Browser Rendering * **HTML Parsing**: Parses the HTML document into a DOM tree. * **CSS Parsing**: Interprets CSS to apply styles. * **Rendering**: Constructs a render tree, lays out elements, and paints them on the screen. * **GPU Rendering**: Uses GPU for faster rendering in modern browsers. #### Post-Rendering * **JavaScript Execution**: Runs any JavaScript on the page. * **User Interaction**: Scripts and network requests can be triggered by user input. # Network Topologies Notes taken from: https\://explained-from-first-principles.com/internet/#nodes-and-links ### Nodes and links <a href="#nodes-and-links" id="nodes-and-links"></a> Nodes (computers) communicate with eachother over links (channels). In graph theory, nodes are called vertices and links are called edges. <figure><img src="/files/BC8ZvsFWda9mshEu0lZn" alt=""><figcaption>Nodes connected by a Link</figcaption></figure> ### Fully Connected Network A fully connected network has direct links from every node to every other node. This scales badly in networking as the number of links grows quadratically with the number of nodes. The formula is `l = n x (n - 1) / 2` In graph theory, this is called a complete graph. <figure><img src="/files/Byu8J7Z9ziwrWerHhdNP" alt=""><figcaption></figcaption></figure> ### Star Network Introducing a central node (**router / relaying node**) reduces the number of links between nodes. In a star network, the number of links scales linearly with the number of nodes. Non-relaying nodes act as communication endpoints. **A star network is centralized**, with the central node exerting control over communication and being a single point of failure. This topology **reduces availability** and may be undesirable for political and technical reasons. <figure><img src="/files/NBOUAPZ4ysLiZ6H1tcMy" alt=""><figcaption>Star network with central node (router) in blue</figcaption></figure> ### Mesh Network Increasing the number of routers (relaying nodes) avoids the drawbacks of centralization. In a mesh network, communication remains possible even if some routers fail. A partially connected network **balances redundancy and scalability.** Mesh networks are typically **preferred for flexibility and reliability**. Critical systems may connect to multiple routers for **increased availability, despite higher costs**. <figure><img src="/files/KNrkIFdMIqzu2JjZwHKk" alt=""><figcaption>Mesh Network with multiple routers</figcaption></figure> # Signal Routing ### **Network Addresses** In networks with routers, nodes need a way to address each other. Even if a router forwards signals to all its links like a hub, nodes must determine if they are the intended recipients of messages. This is achieved by assigning a unique identifier, called a **network address**, to each node. **Each message includes the recipient's identifier, allowing routers to know where to forward communications.** This system works best when addresses are assigned based on the network's geographical structure rather than randomly. <figure><img src="/files/pqETJYVBVxRsnfUgkyJP" alt=""><figcaption>Nodes with addresses according to the router they’re connected to</figcaption></figure> ### **Routing Tables** Routing is the process of selecting a path between two nodes across a network. Routers handle this task using a routing table, which tells the router where to forward communication for each node. For example, in the routing table below, router A forwards messages for node A2 on link 2, regardless of where it receives them from. **Example Routing Table for Router A:** | Destination | Link | Cost | | ----------- | ---- | ---- | | A1 | 1 | 4 | | A2 | 2 | 2 | | B? | 3 | 5 | | B? | 4 | 8 | | C? | 3 | 9 | | C? | 4 | 6 | The router chooses routes based on cost, which could be based on network delay or transmission cost. In this case, router A forwards communications for nodes starting with C on link 4 because it has a lower cost. ### **Forwarding Tables** A forwarding table is a simplified version of the routing table. It only contains the optimal routes, which makes it smaller and faster for lookup, important for low-latency routing. **Example Forwarding Table for Router A:** | Destination | Link | | ----------- | ---- | | A1 | 1 | | A2 | 2 | | B? | 3 | | C? | 4 | ### **Routing Protocols** Routers can fail, and new nodes are added to networks often. To keep routing tables current, routers use routing protocols to communicate changes. For instance, if router A stops hearing from router C, it will update its table to send communications to C through router B. ### **Signal Relaying** Signals can be relayed through a network using either circuit switching or packet switching. ### **Circuit Switching** In a circuit-switched network, a dedicated communication channel is established for the duration of the call. * **Example**: Early telephone networks required a switchboard operator to connect two telephones, creating a closed circuit. * **Advantages**: Signal delay remains constant, and messages arrive in the same order. * **Disadvantages**: Inefficient use of capacity, as other users cannot access the channel when it's idle (e.g., during pauses in conversation). ### **Packet Switching** In a packet-switched network, data is divided into packets, which consist of a header and a payload. * **Header**: Contains delivery information, such as sender and recipient addresses. * Each router has a queue for incoming packets and forwards them based on its forwarding table. * Packet-switching routers do not maintain state, meaning no channels are opened or closed at the routing level. # DNS (Domain Name System) ### **What is DNS?** The Domain Name System (DNS) is a decentralized naming system for devices and services connected to the internet or private networks. It translates human-friendly domain names (like google.com) into machine-friendly IP addresses (such as 192.168.0.1). This allows browsers to locate and load resources on the internet without needing to memorize numerical IP addresses. ### **What layer of the OSI model does it operate at?** **DNS (Domain Name System)** operates at the **Application Layer** (Layer 7) of the OSI model. ### **How Does DNS Work?** DNS resolves human-readable domain names into IP addresses. This process involves four main components: 1. **DNS Recursor**: A server acting as a "librarian" that receives queries from browsers and initiates additional requests to fulfill the DNS query. 2. **Root Nameserver**: The first step in finding an IP address, similar to a library index, directing the query to more specific servers. 3. **TLD Nameserver**: Points to servers associated with the top-level domain (like .com or .net). 4. **Authoritative Nameserver**: The final stop that stores the actual IP address for the requested domain. ### **DNS Lookup Steps:** <figure><img src="/files/QueGXmUbIe65cDCgD8Jo" alt=""><figcaption><a href="https://www.iotforall.com/a-simple-explanation-of-the-domain-name-system">https://www.iotforall.com/a-simple-explanation-of-the-domain-name-system</a></figcaption></figure> 1. A user types in a domain like "example.com." 2. The browser queries a **DNS resolver**. 3. The recursor asks the **root nameserver**, which points to the relevant **TLD server**. 4. The **TLD server** provides the IP address of the domain’s nameserver. 5. The recursor gets the IP address from the authoritative nameserver and returns it to the browser. The browser can now load the web page by sending a request to that IP. Here's another image showing the step by step process: <figure><img src="/files/7QQmaUfePMZoeMecDXfA" alt=""><figcaption><a href="https://bluecatnetworks.com/glossary/what-is-a-dns-server/">https://bluecatnetworks.com/glossary/what-is-a-dns-server/</a></figcaption></figure> ### **Types of DNS Queries:** 1. **Recursive Query**: The DNS client expects a definitive answer (either the record or an error). 2. **Iterative Query**: The DNS server returns the best possible answer or a referral to another server. 3. **Non-Recursive Query**: This occurs when the DNS resolver already has the information cached and returns it without needing further queries. ### **DNS Caching:** To speed up the process, DNS records are cached in various locations: * **Browser Cache**: Modern browsers store DNS records locally for a short time. * **Operating System Cache**: The OS also caches DNS records to avoid repetitive queries. * **ISP Caching**: ISPs store frequently requested DNS records to reduce query time further. In some cases, Cloudflare DNS optimizes this process with infrastructure designed to handle high traffic, such as root DNS servers. # SSL (Secure Sockets Layer) <figure><img src="/files/JqapGMHgxF6hLsEWLblH" alt=""><figcaption><a href="https://www.cloudflare.com/en-ca/learning/ssl/what-is-ssl/">https://www.cloudflare.com/en-ca/learning/ssl/what-is-ssl/</a></figcaption></figure> ## SSL (Secure Sockets Layer) Overview * SSL is an encryption-based security protocol developed by Netscape in 1995. * Ensures privacy, authentication, and data integrity in online communications. * SSL is the predecessor to **TLS (Transport Layer Security)**, the modern encryption standard. ## How SSL/TLS Works 1. **Data Encryption**: * SSL encrypts data transmitted over the internet, making intercepted data unreadable. 2. **Authentication**: * SSL initiates a **handshake** process to verify the identity of both communicating devices. 3. **Data Integrity**: * SSL digitally signs data, ensuring that it hasn’t been tampered with during transmission. ## Importance of SSL/TLS * Originally, data was transmitted in **plaintext**, vulnerable to interception. * SSL encrypts sensitive information (e.g., credit card numbers) to prevent theft. * Authenticates websites to avoid **phishing** attacks and ensures data isn’t altered. ## SSL vs TLS * **TLS** (Transport Layer Security) is the updated version of SSL, introduced in 1999. * SSL 3.0 (last version of SSL) is deprecated due to known vulnerabilities. * Most websites now use TLS, although many still refer to it as **SSL**. ## SSL Certificates * Websites must have an **SSL certificate** (technically a TLS certificate) to implement SSL/TLS encryption. * **Public key** encryption is used for establishing secure communication. * SSL certificates are issued by trusted **Certificate Authorities (CAs)**. ### Types of SSL Certificates * **Single-domain**: Covers one domain (e.g., [www.example.com](http://www.example.com/)). * **Wildcard**: Covers one domain and all its subdomains (e.g., [www.example.com](http://www.example.com/), blog.example.com). * **Multi-domain**: Covers multiple unrelated domains. ### SSL Certificate Validation Levels 1. **Domain Validation (DV)**: * Basic level; verifies control of the domain. 2. **Organization Validation (OV)**: * More involved; CA directly contacts the organization. 3. **Extended Validation (EV)**: * Highest level; requires a full background check of the organization. ## Obtaining an SSL Certificate * **Cloudflare** offers free SSL certificates with easy setup. * Some websites may need additional setup for their origin servers. # TLS (Transport Layer Security) ## Transport Layer Security (TLS) Overview * **TLS** is a widely used security protocol designed for ensuring privacy and data security in online communications. * Primarily encrypts communication between web applications and servers (e.g., web browsers and websites). * Other uses: Encrypting emails, messaging, and VoIP. * **First introduced** in 1999 by the **Internet Engineering Task Force (IETF)**; latest version is **TLS 1.3** (published in 2018). ## TLS vs SSL * **TLS** evolved from the earlier **SSL** protocol (Secure Sockets Layer), developed by Netscape. * **TLS 1.0** started as **SSL 3.1**, but was renamed before release to disassociate from Netscape. * **TLS** and **SSL** are often used interchangeably due to their close relationship. ## TLS vs HTTPS * **HTTPS** is HTTP combined with **TLS encryption**. * Any website using HTTPS is utilizing **TLS** to encrypt communication. ## Why Should Businesses Use TLS? * **TLS encryption** protects against data breaches and cyberattacks. * Major browsers (e.g., **Google Chrome**) warn users about non-HTTPS sites. * **HTTPS padlock icon** is a sign of a secure, TLS-protected connection. ## TLS Components 1. **Encryption**: Hides transferred data from third parties. 2. **Authentication**: Verifies the identity of communicating parties. 3. **Integrity**: Ensures data has not been altered during transmission. ## TLS Certificates * To use TLS, a website must have a **TLS certificate** (commonly referred to as an **SSL certificate**). * Issued by **certificate authorities (CAs)** to the domain owner. * Contains the domain’s ownership details and **public key** (used for encryption). ## How TLS Works #### TLS Handshake * The handshake process establishes a secure connection between the user's device (client) and the web server. 1. **Specify TLS version**: (e.g., TLS 1.0, 1.2, 1.3). 2. **Choose cipher suites**: Set of algorithms for encryption. 3. **Server authentication**: Verifies server’s identity using the **TLS certificate**. 4. **Generate session keys**: For encrypting messages after the handshake. * **Public key cryptography**: Uses a public key to decrypt data, while only the server’s private key can encrypt it. * After encryption and authentication, the data is signed with a **Message Authentication Code (MAC)** for integrity. ## Performance Impact of TLS * Modern **TLS versions** (e.g., **TLS 1.3**) have minimal impact on web performance. * **TLS False Start** and **Session Resumption** mitigate potential latency by speeding up the handshake. * **TLS 1.3** improves speed with a 1-round-trip handshake and zero round trips for previously connected users. ## Implementing TLS * **Cloudflare** offers free TLS/SSL certificates. * Alternatively, businesses can acquire and install an **SSL certificate** from a **certificate authority** on their servers. # Process ## Requirements #### **Key Simplified Workflow** 1. Write 3–5 **Functional Requirements**. 2. Write 3–5 **Non-Functional Requirements** with specific metrics. 3. Skip detailed capacity math unless it's crucial for design decisions. *** ### **Functional Requirements (Core Features)** * **Goal**: Identify the top 3 features your system must have. * **Approach**: * Think about what **users** or **clients** will do with the system. * Use **"Should be able to..."** statements to frame these features. * Example: * For Twitter: "Users should be able to post tweets." * For a Cache: "Clients should be able to read items." 💡 **Tip**: Prioritize only the most critical features. Skip minor details or edge cases. *** ### **Non-Functional Requirements (Qualities)** * **Goal**: Define key qualities your system should meet (e.g., speed, reliability). * **Approach**: * Use **"The system should..."** statements, with specific metrics when possible. * Example: * For Twitter: "The system should render feeds in under 200ms." * For a Cache: "The system should tolerate one node failure without data loss." 💡 **Checklist for Non-Functional Needs**: * **CAP Theorem**: Consistency or Availability? * **Scalability**: Will traffic grow, and how (e.g., bursty traffic)? * **Latency**: Which operations need to be fast (and how fast)? * **Durability**: Is data loss acceptable? (Critical for banking; less so for logs.) * **Fault Tolerance**: How many failures should the system survive? *** ### **Capacity Estimation (Optional)** * Skip this unless **calculations affect your design**. * Example: * For TopK trending topics: Estimate the number of topics to decide between using a single machine or sharding data. 💡 **Tip**: Explain to the interviewer that you'll do math only if necessary, focusing instead on relevant design trade-offs. *** ## **Core Entities** **Goal**: Identify the main objects in your system to build your foundation. * **Approach**: * **Who are the actors?** (e.g., users, clients). * **What nouns/resources fulfill functional requirements?** * Jot down a **bulleted list** of key entities. This is a **first draft**, so avoid overcomplicating with details like relationships or fields. * **Example (Twitter)**: * **User** * **Tweet** * **Follow** 💡 **Tip**: Use clear, intuitive names for entities to make it easier for everyone to follow. *** ## **API or System Interface** **Goal**: Define how the system communicates with users or clients (its **contract**). * **Decision**: Choose a communication protocol: * **REST API** (default for simplicity): Uses standard HTTP verbs (GET, POST, etc.). * **GraphQL**: Use only if clients need fine-grained control over the data returned. * **Wire Protocol**: Define a message format for WebSocket or TCP connections. * **Steps**: * Base your endpoints on **core entities**. * Write endpoint definitions that map to **functional requirements**. * **Example (Twitter REST API)**: * **POST /v1/tweet** * Body: `{ "text": string }` * Authenticated user ID inferred from token. * **GET /v1/tweet/:tweetId** → Returns a **Tweet**. * **POST /v1/follow/:userId** → Authenticated user follows another user. * **GET /v1/feed** → Returns a list of **Tweet\[]**. 💡 **Security Tip**: Avoid putting sensitive data like `userId` in the body or query params when it can be derived securely from headers (e.g., auth tokens). *** ## **High-Level Design** **Goal**: Build a simple architecture to meet your system’s functional requirements and API design. * **Focus**: * **Components**: Servers, databases, caches, message queues, etc. * **Interaction**: Draw arrows to show how components communicate. * **Keep It Simple**: Start with basic components and focus on functional requirements. * **Step-by-Step**: 1. **Identify core components**: What do you need for your system to work? For example, a web server, a database, etc. 2. **Flow of data**: Discuss how data flows from one component to another, focusing on state changes during each request. 3. **Start with API endpoints**: Build your design around these endpoints, adding complexity only when necessary (e.g., caching, queuing). * **Visual Tips**: * You don’t need to document every single detail in your schema. Focus on **important fields** that affect your design. * As you draw your architecture, talk through your thought process, explaining how each component interacts. * **Example (Twitter)**: * POST `/v1/tweet` → Send to server → Store in database. * GET `/v1/tweet/:tweetId` → Fetch from database → Return to client. * POST `/v1/follow/:userId` → Update user’s following list → Store in database. * GET `/v1/feed` → Use cached feeds or query database for timeline. *** ## **Deep Dives** **Goal**: Harden your design by addressing non-functional requirements, edge cases, and bottlenecks. * **Focus**: 1. **Non-functional requirements**: Scalability, performance, fault tolerance. 2. **Edge cases**: Handle unexpected conditions like failed API requests or database downtime. 3. **Bottlenecks**: Identify and improve performance issues. 4. **Probes**: Address any interviewer feedback or questions during this section. * **Proactive Discussion**: * **Junior candidates**: Your interviewer will guide you, pointing out places to improve. * **Senior candidates**: Lead the discussion by identifying potential problems and discussing solutions. * **Example (Twitter)**: * **Scalability**: Horizontal scaling, caching, and database sharding to handle >100M DAU. * **Feed Latency**: Discuss **fanout-on-read** vs **fanout-on-write** strategies and caching to minimize latency when fetching user feeds. * **Collaboration Tip**: Don’t dominate the conversation. Allow space for the interviewer to probe your design. Their questions may provide valuable insights or signal areas they want to focus on. # Kafka The following notes are all taken from reading [HelloInterview](https://www.hellointerview.com/learn/system-design/deep-dives/kafka): ## Overview When an event happens, the producer creates a message (also called a record) and sends it to a Kafka topic. Each message includes a required **value** field and three optional fields: 1. **Key**: Determines which partition the message goes to. 2. **Timestamp**: Helps to order messages within a partition. 3. **Headers**: Key-value pairs, similar to HTTP headers, used to store metadata about the message. <div align="center"><figure><img src="/files/YPiJCnpu7cbw5TOxUVgw" alt="" width="338"><figcaption>Kafka Message Structure</figcaption></figure></div> **Partition Assignment:** Kafka assigns messages to partitions based on their key. If a message has no key, it uses a round-robin or another set rule. Messages with the same key always go to the same partition, keeping them in order. **Broker Selection:** Kafka identifies which broker handles the partition using cluster data. The producer then sends the message directly to that broker. <figure><img src="/files/dhAZKxcFFOzxo2kR1N2h" alt=""><figcaption>Kafka Architecture</figcaption></figure> ## Terminology ### **Kafka Cluster** * Composed of multiple **brokers** * More brokers = higher scalability for storage and client handling. ### **Broker** * The servers (physical/virtual) that hold the "queue". * Stores data and manages client requests. ### **Partition** * Ordered, immutable sequence of messages, like a log file. * Key for scaling, as partitions enable parallel message consumption. ### **Topic** * Logical grouping of **partitions**. * Used for publishing and subscribing to data. * Supports multiple producers writing data simultaneously. ### **Topic vs Partition** * **Topic**: Logical organization of messages. * **Partition**: Physical organization of messages (can span multiple brokers). ### **Producers and Consumers** * **Producers**: Write data to topics. * **Consumers**: Read data from topics. * Kafka provides APIs for both but leaves message creation/processing to developers. ### **Message Queue vs Stream** * **Message Queue**: Consumers acknowledge messages after processing. * **Stream**: Consumers process messages without acknowledgments, enabling complex processing. # Advanced Topics For scaling Kafka, focus on partitioning (key choice and number of partitions) and adding brokers. For fault tolerance, use replication and track consumer offsets. To improve performance, batch and compress messages, and always think about efficient partitioning. ## **Kafka Broker Constraints**: * A single broker can store **\~1TB** and handle **\~10,000 messages/sec** (depends on hardware). * Keep Kafka messages small **(<1MB)** for optimal performance; Kafka is not for storing large files. * Use Kafka for small messages like pointers (e.g., store large videos in S3, not in Kafka). ## Scalability * **Horizontal Scaling**: Add more brokers to distribute load. Ensure enough partitions to utilize all brokers. * **Partitioning Strategy**: Choose a good key for partitioning (e.g., ad ID). A bad key can cause "hot partitions" (overloaded). * Use random partitioning, salting (adding randomness), or compound keys to handle hot partitions. ## Fault Tolerance & Durability * **Replication**: Each partition is replicated to ensure durability. The replication factor (e.g., 3) defines how many replicas exist. * The replication factor should not exceed the total number of brokers in your cluster. * **Rule of Thumb**: * Ensure the replication factor is less than or equal to the number of brokers. * **Producer Acknowledgments (acks)**: Set `acks=all` for maximum durability—ensures all replicas confirm receipt of a message. * **Consumer Recovery**: Offsets are tracked to ensure consumers can pick up where they left off if they crash. Rebalancing happens automatically if a consumer fails. ## Errors & Retries * **Producer Retries**: Kafka producers automatically retry sending failed messages with a configurable number of attempts. * **Consumer Retries**: Kafka doesn't handle retries for consumers natively, but you can set up a separate "dead letter queue" (DLQ) for retrying or logging failed messages. ## Performance Optimizations * **Batching**: Send messages in batches to reduce overhead. Adjust `maxSize` and `maxTime` for better throughput. * **Compression**: Compress messages (e.g., GZIP) to improve speed by reducing message size. * **Partitioning Strategy**: Ensure even distribution across partitions for better parallelism and throughput. ## Retention Policies * Kafka allows setting a retention period for messages via `retention.ms` (time-based) or `retention.bytes` (size-based). The default is 7 days or 1GB. * If you need longer storage, adjust retention settings—but be mindful of storage costs and performance trade-offs. # URL Shortener When discussing the system design of a URL shortener like TinyURL or Bit.ly, ensure you cover the following key aspects. Structuring your notes around these will help you stay organized during the interview. *** ## Requirements ### **Functional Requirements** * Shorten a given URL and return a unique, shortened version * Redirect users to the original URL when they use the shortened URL. * Custom alias support * Optionally, allow expiration and analytics (clicks, user analytics, etc.). ### **Non-Functional Requirements** * **Focus on availability over consistency**: The service should always be up. * **Low Latency**: Redirection should be fast. * **Scalability**: Support millions or billions of URLs. * **Durability**: Store URLs reliably to prevent data loss. *** ## **Estimation and Capacity Planning** * **QPS (Queries Per Second)**: Estimate reads and writes (e.g., 10:1 ratio). * **Storage Requirements**: Assume average URL length and calculate storage needs (e.g., 1 billion URLs). *** ## Entities * Original URL * Shortened URL * User *** ## **API Design** * **POST `/shorten`** -> returns a shortened URL. * { originalUrl, customAlias?, expirationTime? } * **GET `{shortUrl}`**-> Redirects to the original URL. * **HTTP 302 (temporary redirect)** instead of HTTP 301 (permanent redirect which browser caches) * Optional APIs for stats or management: * **GET `/stats/{shortUrl}`**. **b) Database Schema** * `ShortUrl (short_key, original_url, created_at, expiration_date)` * Index on `short_key` for fast lookups. **c) URL Shortening Logic** * **Short Key Generation**: * Use a **Base62** encoding scheme (characters `a-z`, `A-Z`, `0-9`) to keep keys short and human-readable. * A 6-character Base62 key can represent over 56 billion unique URLs (62⁶). * **Counter/Sequence**: Increment a global counter and encode the value in Base62. * A single Redis instance on modern hardware can handle **80,000 to 100,000 operations per second (OPS)** for simple commands like `INCR`, assuming: * Redis is running on dedicated high-performance hardware. * The network is low-latency and high-bandwidth. * There are no significant memory or disk I/O bottlenecks. * Handle **collisions** in case of duplicate keys. **d) Data Storage** * Choose a storage system based on scale: * **Relational Databases**: MySQL/PostgreSQL for small-scale systems. * **NoSQL Databases**: DynamoDB, Cassandra, or MongoDB for large-scale systems. * In-memory caching (Redis or Memcached) for frequently accessed data. **e) Redirection** * Use **HTTP 301 (Moved Permanently)** or **302 (Found)** for redirection. *** ## **Scaling the System** **a) Read/Write Patterns** * Reads will dominate writes (many more redirections than shortenings). * Optimize for high-read throughput using caching. **b) Partitioning and Replication** * Use database replication for durability, fault tolerance and to allow scaling reads horizontally. * Partition data by the short key to distribute load (consistent hashing). **c) Caching** * Use Redis or Memcached to cache mappings of frequently accessed `short_key -> original_url`. **d) Rate Limiting** * Implement rate limiting to prevent abuse (e.g., spamming the shorten API). *** ## **High-Level Architecture** <figure><img src="/files/b1tpY7uEIx7dfdRHz1Ou" alt=""><figcaption></figcaption></figure> *** ## **Trade-offs and Challenges** * **Key Length**: Shorter keys are user-friendly but increase the chance of collisions. * **Consistency**: Balancing consistency and availability (CAP theorem). * **Redundancy**: Ensure data is not lost due to server failures. * **Performance**: Optimize key lookup and redirection latency.