Best practices

Intro

An undeniable, intricate bond between the infrastructure and the application logic. The architecture of your application is fundamental to its functionality, significantly influencing its operational efficiency and the quality of the developer experience it delivers.

This developer experience might not seem to matter at first glance, but like everything in this world small things add up over time and become large things.

With this in mind, I've tailored Astra to align with a variety of best practices I've gathered over my years of experience in developing both infrastructure and applications.

Adherence to everything on this list isn't mandatory, but following it ensures that Astra operates optimally and you avoid unnecessary complications. Down the line the hope is that this will lead to fewer big things you'll have to deal with.

Lastly, Astra is meant to be opinionated; as standardization means consistency. So while the best practices listed below might not be the objective right way to set things up for everyone, we've found they tie together well with our values and make for a smooth ride.

These best practices are divided into three categories:

  • Required: Without these, Astra's functionality will be severely limited.
  • Recommended: Skipping these may hamper Astra's functionality, but it will remain mostly usable.
  • Suggested: Astra's functionality will be mostly unaffected if these are not followed.

Required: Astra Requires the Use of Containers

Astra is designed with a cloud-native approach, assuming familiarity and use of containers as integral to its operations. Its paradigms and most features depend on container usage, which is essential for leveraging the full capabilities of Astra.

This document outlines processes and best practices that are centered around container utilization.

If you're new to working with containers or have any questions, Astra's support is ready to assist you!

Required: Environment Variables and Application Configuration

The topic of handling application configuration can prompt a wide range of answers, with each being valid in its context. Astra adopts a method that aligns with the cloud-native principles outlined in the 12-factor application philosophy.

This means your application should be designed to accept environment variables as a primary means to receive configuration values. Here's a closer look at this approach:

Reading Everything into an Object/Struct

Your configuration needs to be accessible across various parts of your code. This is typically achieved by loading the configuration into a single object or struct. This object can then be shared among different parts of the application that require access to these configuration values.

Below is a simplified example (note: this is illustrative and not actual working code):

// First, define your configuration struct
type AppConfig struct {
    DatabaseURL   string `json:"databaseUrl"`
    ServerPort    int    `json:"serverPort"`
    EnableLogging bool   `json:"enableLogging"`
    APIKey        string `json:"apiKey"`
}

// Then, create a function to read configuration from a file
// and populate the AppConfig struct, returning the populated struct.
func LoadConfiguration(file string) (AppConfig, error) {
    var config AppConfig
    configFile, err := os.Open(file)
    if err != nil {
        return config, err
    }
    defer configFile.Close()

    jsonParser := json.NewDecoder(configFile)
    err = jsonParser.Decode(&config)
    return config, err
}

Create a Hierarchy

I recommend establishing a hierarchy for parsing your application's configuration. The recommended order is as follows:

defaults(optional) -> config(optional) -> env vars[required] -> flags(optional)

This hierarchy implies:

  1. By default your application should populate the struct with pre-set hardcoded defaults. Defaults are a great way to kickstart your development for newer developers.

  2. Next, your application should search for configuration files in specified locations. While not used in deployments, configuration files are valuable during development, offering a convenient alternative to repeatedly setting environment variables.

  3. Subsequently, your application should prioritize environment variables. These are always prefixed with your application's name, ensuring a direct match with the configuration struct fields. For example, an application named "testbed" would look for the following environment variables:

    • TESTBED_DATABASE_URL
    • TESTBED_SERVER_PORT
    • TESTBED_ENABLE_LOGGING
    • TESTBED_API_KEY

🪧 Environment variables are essential, as container orchestrators rely on them to inject configuration into your application container.

  1. Lastly, while optional, your application can also accept flags to adjust its configuration.

Each level of the hierarchy has the potential to overwrite values from the previous level, but only if it provides a value for a given key.

For example:

  1. DatabaseURL defaults to localhost:5432.
  2. The configuration file specifies test.db.staging:5432.
  3. An environment variable sets this to db.prod:5432.

Given this setup, the effective value would be db.prod:5432 because environment variables are prioritized over defaults and config files in the hierarchy.

However, if you launch your application with the flag --database_url="db2.prod:5555", this flag's value (db2.prod:5555) overrides all others due to its higher precedence.

Some Other Important Points

Use an Established Library

This configuration pattern is well-established. Typically, your chosen programming language will offer one or several libraries that facilitate easy setup of this structure.

Should you require additional guidance on implementation, Astra support is available to assist.

Developer Experience is Important

  • Enhancing the developer experience, though a secondary objective, is crucial. Writing application code (and documentation) in a way that allows newcomers to easily engage with the project adds immense value.

  • Defaulting to non-critical values wherever possible enables new developers to start working on the project without hurdles. Ideally, running the application locally should "just work." Configurations can then be adjusted by the developer to suit different environments or specific development needs.

  • Getting to this point can require some planning, for example: certs. You might be thinking "if my application needs certs I'll have to have new developers generate them before they can start developing the application." Avoid this for as long as you can. With a little ingenuity most of this can be offered to the developer free of cost. Embedding a localhost cert into the binary and creating a configuration where the localhost certs can only be turned on in development is an easy way to give a better developer experience around writing application code for newer developers.

  • While using sensible defaults is advisable, not all settings will have obvious default values. In cases where essential configurations lack defaults, the application should explicitly fail to start, signaling the absence of necessary values. In the spirit of lowering the bar to entry your project can maintain a command that allows people to generate a configuration file where the missing values dev values can be inserted for ease of use.

Challenges with Expressing Configuration Through Environment Variables

  • Nesting Complexity: One common issue with environment variables is their inability to naturally express nested configurations. Given that underscores are typically used to delineate names within the same hierarchical level, it can become unclear how to represent multi-level configurations. For instance:

    type AppConfig struct {
  Database *Database `json:"database"`
}

type Database struct {
  SomeURL string `json:"some_url"`
}

Fortunately, many libraries supporting environment variables have mechanisms to accommodate nesting, often through the use of additional prefixes. For example, an object like database.url might be represented as TESTBED_DATABASE__SOME_URL in environment variables.

  • Complex Structured Types: Another challenge arises when attempting to express configurations that involve complex structured types within environment variables. The recommended approach for such cases is to employ a serialization format, commonly JSON. Consider the following example:

    type AppConfig struct {
// Assuming this is a complex object that is difficult to directly map to environment variables.
  EmailSettings *EmailSettings `json:"email_settings"`
}

type EmailSettings struct {
  SMTPHost string `json:"smtp_host"`
  SMTPPort int    `json:"smtp_port"`
}

    Then within the env var we can just put a JSON object:

    # It should be noted that this isn't a very good example as it can be well represented by env vars but the general idea stands.
export TESTBED_EXTENSION_SETTINGS="{'email_settings': {'smtp_host': 'smtp.example.com', 'smtp_port': 587}}"

Handling secrets poses a unique challenge, as they cannot be managed in the same manner as other configuration data due to their sensitive nature. However, Astra integrates secrets management seamlessly, ensuring they are as accessible as other configurations without sacrificing security.

For an introduction to managing secrets with Astra, refer to the guide on using Astra for secret management.

Astra simplifies the process of updating secrets by assigning a new identifier to each updated secret. This approach allows for straightforward updates and the ability to revert to a previous version if necessary.

🪧 I wont give you the security wag of the finger here, but it's important to remember that these are secrets and they're only as good as their weakest link. Be careful about how you treat and share them.

Required: Using Terraform for Out of Astra Infrastructure Changes

While Astra endeavors to fulfill a broad range of infrastructure requirements, it acknowledges the impossibility of catering to every possible need. As such, you may find yourself seeking to customize or exert more control over your infrastructure.

Astra facilitates this by exporting its operations to Terraform configurations, allowing for modifications as per your preferences.

The recommended approach involves treating your Terraform files with the same rigor as application code. Early in your startup's lifecycle, adhere to a straightforward workflow of:

approve -> apply -> merge
  • Approve: Run any changes to your Terraform through the normal software review process. Create a PR, get it reviewed, the works.
  • Apply: Once approved, communicate with your team you're about to apply your changes and then run terraform plan to verify they are indeed the changes you would like to run. After you're convinced you have the correct changes run terraform apply, check again, and hit the big red button!
  • Merge: Finally merge the pull request so your changes are immortalized in git's history.

🪧 You can find additional Terraform Best Practices here.

Suggested: Keep database migrations simple

Database migrations can be daunting since touching your database with any schema change can result in bad things.

I've had success with the following process:

  • Utilize a Migration Library: Choose a migration library that seamlessly integrates into your development workflow. While some libraries offer command-line interfaces (CLIs), prioritizing simplicity is key. Opt for libraries that can be directly incorporated into your codebase, reducing the need for additional management overhead.

  • Automate Migrations at Application Startup: Implement your migration process as an integral part of your application's startup routine. This approach ensures that migrations are executed consistently and are version-controlled within the same repository as your application code, adhering to standard coding practices. Such integration simplifies the deployment process and ensures that migrations are consistently applied across environments.

  • Prefer Rolling Forward: Migration tools typically support both forward ("up") and backward ("down") migrations. While having the option to revert changes ("down") is valuable, it's advisable to focus on rolling forward in case of issues. If a migration introduces a problem, address it by applying a new "up" migration. This strategy maintains the integrity of the migration path, ensuring a clear and linear history of database schema changes, and facilitates easier understanding of the database's evolutionary path.

  • Test Migrations with Production-Scale Data: It's common practice to test migrations on development databases, which may not reflect the complexity or scale of production environments. This discrepancy can lead to unanticipated issues, such as prolonged table locks during migration, potentially impacting application availability. To mitigate this risk, simulate migrations using datasets of similar size and complexity to those in production, ensuring that your migration strategy is robust and reliable under real-world conditions.

Suggested: Streamlining Deployments, Releases, and Versioning

Deploying in a structured fashion can be difficult. Astra helps a ton by taking care of some of the harder parts of what it means to deploy an app expected to be used 24/7, but there is so much more to be done. Some small notes on how you should be thinking about your deployments:

  • Treat Deployments as Immutable Snapshots: Consider each deployment a fixed snapshot of your application at a specific moment. Avoid altering settings or configurations post-deployment. Instead, incorporate any changes, including environment variable adjustments, in a new version of the application and proceed with a standard deployment process. This practice ensures consistency, making it simpler to troubleshoot by providing a stable reference point for the production environment.

  • Adopt Software Versioning: Although immediate implementation isn't necessary, developing a routine for versioning your releases is beneficial. Semantic versioning (semver) is recommended for its clarity and systematic approach to version increments (patch, minor, and major). This not only fosters a culture of precision but also makes it straightforward to identify what version is currently in production.

  • Prepare for Rollbacks: Stability hinges on the ability to revert changes when new deployments introduce issues. With Astra, rolling back is streamlined, thanks to the tracking of application changes via the manifest file. Being prepared to rollback ensures that any destabilizing changes can be quickly undone, maintaining the integrity and stability of your application.

    Incorporating these practices into your deployment and release strategy enhances the manageability and reliability of your software, aligning with industry standards for continuous delivery and operational excellence.

Suggested: Implementing Telemetry

As your application scales, the need for comprehensive telemetry to gain insights into its operations becomes increasingly critical. Here are two fundamental recommendations to establish a robust telemetry framework:

  • Use OpenTelemetry: OpenTelemetry has standardized application telemetry so that it's easy to build tools that collect and process it. Most languages have libraries which adhere to these standards. Using opentelemetry means you aren't tied forever to a single telemetry paradigm. If you start with datadog, but want to switch later down the line to save costs(a common scenario) it's easy(ier) to switch.

  • Use a logging library which has multiple outputs: Reading JSON is for computers, not for humans. When your application is in development mode, use plaintext and colors to make parsing logs easy to read. When you're in production and your logs are being ingested by a tool, use JSON.

Suggested: Opting for ECR as Your Container Repository

Incorporating your application into a Docker container necessitates a reliable repository for storing and managing these containers. Astra facilitates this by integrating with AWS Elastic Container Registry (ECR), offering a seamless solution for your container orchestration needs.

To locate your ECR registry address, utilize the service get command provided by Astra. This address is pivotal for managing your Docker images. For detailed instructions on uploading containers to ECR, consult the AWS documentation on pushing Docker images to ECR.

A straightforward example to authenticate, tag, and push your Docker images to ECR might look like this:

# Set Astra authentication for the staging environment
~|⇒ astra auth set staging -r staging_admin

# Log in to AWS ECR
~|⇒ aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin 3335161047939.dkr.ecr.us-east-2.amazonaws.com/testbed

# Push your containers
~|⇒ docker push 335161047939.dkr.ecr.us-east-2.amazonaws.com/testbed:latest
~|⇒ docker push 335161047939.dkr.ecr.us-east-2.amazonaws.com/testbed:2.6.3

🪧 As mentioned in the section on deployments and versioning, taking time to version your application is a good habit. This means that tagging your docker containers with the appropriate version is very important.