AWS IAM (Identity and Access Management) roles are the backbone of secure access control in cloud environments. However, managing cross-account IAM roles can quickly become challenging or confusing, especially if you’re missing critical permissions like sts:AssumeRole. Additionally, an incorrect or missing ExternalId can prevent roles, such as those used by third-party auditors or monitoring systems, from being assumed, as it is a key requirement for establishing cross-account trust.

This guide will walk you through common scenarios of cross-account communication, highlight potential pitfalls, and provide solutions with detailed flowcharts and examples.

Why Cross-Account IAM Roles Matter

In large organizations, production accounts are often separated by business function for security, billing, and compliance reasons. For example:

• Analytics Account

• Data Science Account

• Centralized Logging Account

Cross-account IAM roles enable secure interactions across these environments, ensuring proper access control and permissions while preventing overly permissive configurations.

Real-Life Analogy: Checking into a Hotel

Think of cross-account access like checking into a hotel:

1. Establish Trust:
   The receptionist verifies your reservation and identity.

   

   • In AWS, this is the trust policy ensuring the target account trusts the source account.

2. Grant Permissions:
   You receive a key card programmed for your room.

   

   • In AWS, this is the sts:AssumeRole permission, allowing temporary access to the role.

3. Access Resources:
   You use the key card to access your room and amenities.

   

   • In AWS, this corresponds to resource-specific permissions like accessing an S3 bucket or DynamoDB table.

This simple analogy illustrates how cross-account access works in AWS and why each step is critical.

What is a Principal?

In AWS, a Principal is an entity that can make requests to AWS services. A Principal must have permissions, either explicitly or implicitly granted, to access AWS resources.

Types of Principals in AWS

1. IAM Users:

   • A user created within an AWS account, typically representing an individual or application.

   • Example: arn:aws:iam::123456789012:user/Alice.

2. IAM Roles:

   • Temporary credentials associated with a specific role, allowing access to AWS resources. Roles can be assumed by users, applications, or other AWS services.

   • Example: arn:aws:iam::123456789012:role/EC2AccessRole

3. AWS Services:

   • Services like Lambda, EC2, or ECS that need access to other AWS resources to function.

   • Example: arn:aws:iam::123456789012:role/service-role/LambdaExecutionRole.

4. Federated Users or Applications:

   • External users authenticated by an identity provider (e.g., SAML, OpenID Connect) or by using AWS STS tokens.

   • Example: A user accessing AWS through a corporate identity provider.

Why is "Principal" Important in Cross-Account Access?

When setting up cross-account access, the first step is to identify the source Principal (the requester) and the target Principal (the resource owner). Knowing which entity assumes roles, accesses resources, and grants permissions helps clarify how AWS IAM policies and trust relationships function.

Understanding the Workflow

The core of cross-account IAM role configuration involves the following steps:

1. Establish Trust: Make sure the target account trusts the role from the source account.

2. Grant Permissions: Provide the necessary permissions for the source account to assume the role.

3. Access Resources: Ensure the source account has the permissions needed to interact with resources in the target account.

Below is a flowchart illustrating the decision-making process to consider when setting up IAM roles.

Examples of Cross-Account Scenarios

Scenario 1: Analytics Account Accessing Data in Data Science Account

• Accounts Involved:

  • Analytics Account: Runs business intelligence (BI) tools like Tableau or AWS QuickSight.

  • Data Science Account: Stores cleaned and processed data in S3 for model training.

• Steps:

  • Add Analytics Account's IAM Role to the S3 bucket policy in Data Science Account.

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Principal": {
              "AWS": "arn:aws:iam::ACCOUNT_ANALYTICS:role/BIReaderRole"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::data-science-bucket/*"
          }
        ]
      }
    

    The arn:aws:s3::: format for S3 ARNs is unique because S3 buckets are global resources, not account-specific. Unlike most AWS services, which include the account ID in their ARNs, S3 does not.

  • Ensure `sts:AssumeRole permission for the Analytics Account's IAM Role.

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::ACCOUNT_DATA_SCIENCE:role/DataAccessRole"
          }
        ]
      }
    

Scenario 2: Data Science Account Accessing Centralized Logging

• Accounts Involved:

• Data Science Account: Runs machine learning models that generate logs for debugging.

• Centralized Logging Account: Collects and stores all logs in CloudWatch.

Steps:

• Grant the Data Science Account IAM Role access to the CloudWatch Logs in Centralized Logging Account.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNT_DATA_SCIENCE:role/LogWriterRole"
      },
      "Action": "logs:PutLogEvents",
      "Resource": "arn:aws:logs:region:ACCOUNT_LOGGING:log-group:/data-science-logs:*"
    }
  ]
}

• Add sts:AssumeRole permission for the Data Science Account IAM Role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::ACCOUNT_LOGGING:role/LogAccessRole"
    }
  ]
}

Common Pitfalls in Cross-Account Access

1. Missing sts:AssumeRole Permission:
Even with correct trust policies, access fails if this permission is not granted.

2. Misconfigured Resource Policies:
Ensure the target resource (e.g., S3 bucket or RDS) explicitly allows the requesting account.

3. Incorrect Role ARN:
Double-check ARNs in all trust and IAM policies to prevent mismatched configurations.

Conclusion

AWS IAM roles are powerful, but cross-account setups can be tricky without proper configurations. Ensuring trust policies, sts:AssumeRole permissions, and resource policies are aligned is critical for secure and seamless communication across production accounts. Additionally, always verify the use of ExternalId in scenarios where it is required, such as for third-party auditors or external monitoring tools, to prevent unexpected access issues.

By following the workflow and using the examples above, you can navigate complex use cases and avoid common pitfalls, ensuring a secure and well-configured cloud environment.

References

• Using IAM Roles to Delegate Permissions to Third Parties

• IAM Policies and Conditions: Using ExternalId

• IAM Roles Trust Policy Examples

One of the standout teachings was the distinction between Jobs and Deployments. While both are controllers in Kubernetes designed to manage pods, they cater to different use cases. Jobs are suitable for one-off tasks, ensuring a particular task reaches completion. In contrast, Deployments ensure a specified number of pod replicas are consistently running. Grasping this difference proved pivotal in multiple projects, enhancing task performance and completion.

The authors also shed light on Kubernetes' robust rollback mechanisms. It's more than just retracing to a prior version; it's about appreciating the nuances and potential trip-ups. Given Kubernetes' inherent ability to revert deployments, I felt bolstered when initiating changes, assured that a trustworthy safety net was at my disposal should things veer off track.

Moreover, the strategic utility of labels and annotations was a revelation. Labels aid in the organization and resource selection, whereas annotations enable you to store extra metadata to elucidate the behavior of tools or libraries. This not only streamlines organization but offers a clearer picture when several teams or projects utilize the same cluster.

In summation, "Kubernetes: Up and Running" by Kelsey Hightower, Brendan Burns, and Joe Beda augmented my grasp of numerous Kubernetes elements and practices. It remains a preferred resource, offering clarity during moments of doubt.

Here are some key takeaways:

1. Software as Craftsmanship: Just like a blacksmith shapes metals or a carpenter creates furniture, a software developer builds applications. It's not just about the tools but the skills. In the realm of craftsmanship, a blacksmith meticulously hones their skills to shape metals, while a carpenter refines their techniques to produce elegant furniture. Similarly, a software developer artfully molds code to birth applications. As time unfurls, this code matures and becomes more polished, mirroring the progression seen in a craftsman's masterpieces. Peer learning, knowledge sharing, and collective problem-solving act as the linchpins for improvement. And at the heart of it all is pride—just as craftsmen revel in their creations, developers too should take immense pride in their code.

2. DRY Principle: It means "Don't Repeat Yourself." Avoid using the same code everywhere. Think of it like this: if you use the same piece of code in many places, you'll have a tough time making changes later on. By avoiding repeated code, you make your work easier to update and less prone to mistakes.

3. Orthogonality: Make sure different parts of software can work on their own. Think of a set of toy building blocks. If you pull one block out, you wouldn't want the whole tower to fall, right? Keeping things orthogonal is like having different departments in a store; if the toy section is closed, it doesn't mean the whole store shuts down.

4. Automate: Reduce repetitive tasks. Use tools that can do the work for you, like a dishwasher cleans dishes without you washing them by hand. Instead of doing repetitive tasks manually, we use tools that do the job for us. It's all about making things easier and more efficient.

5. Code with Quality: Make your code clear, like a well-written story. Instead of using names like "x" or "y" for variables, use names that tell you what they are, like "totalPrice" or "userName". It's all about making sure that if someone else looks at your work, they can easily get the whole picture.

6. Decoupling: Keep components independent. For instance, in software systems, rather than having one component directly call and depend on another, we might use a queue. This way, even if the email system is down or being updated, the website can continue to function normally.

7. Stay Updated: The tech world is always changing, just like fashion trends. Keep up with the latest to stay relevant. It's about taking the time to learn about the newest tools, languages, or ways of doing things.

8. Testing: Before sharing your software, test it. Automated testing is an even better helper. Instead of manually checking every detail, you have tools that run through your code, looking for any issues.

9. Flexibility: Be ready to adapt. Being flexible means being ready to adapt when needed. It's like having a backup plan for that picnic. With a flexible approach, you can adjust your code or strategy, ensuring your software stays relevant and effective no matter what comes its way.

10. Estimations: Plan realistically. When figuring out how long a project or task will take, it's essential to be realistic. By giving realistic estimates, you ensure a smoother journey in coding, with fewer surprises.

Get the book on Amazon

The overarching message is clear: prioritize clean code. But what does "clean" really mean in the context of coding? Here are some of the defining elements and key takeaways from the book:

1. Readability: At its core, code is a form of communication. Clean code should read like well-written prose. This ensures that any developer, whether they wrote the code or not, can understand its purpose and functionality with ease.

2. Meaningful Names: Naming is pivotal. A variable named x tells you nothing, but userAge is descriptive. The names you choose should reveal intent and make the code's functionality evident.

3. Functions Should Do One Thing: A well-written function has a single responsibility. By ensuring each function is concise and serves one purpose, you maintain clarity and reduce potential points of failure.

4. Error Handling: Proper error handling is an essential aspect of clean code. It's not just about catching errors but doing so in a manner that doesn't introduce more complexity or obscure the main logic.

5. Avoid Duplication: The DRY (Don't Repeat Yourself) principle isn't just about reducing the amount of code but ensuring that there's a single, authoritative source of truth in the system.

6. Comments Aren't Always Good: Contrary to popular belief, comments can sometimes be detrimental. If you're using comments as a crutch to explain complex code, it's often a sign that the code itself isn't clean or clear.

7. Limit Function Arguments: The fewer arguments a function has, the easier it is to test and use. If a function is expecting numerous inputs, it might be doing too much.

8. Embrace Testing: Automated tests provide a safety net, ensuring that as you refactor and clean your code, you aren't introducing new bugs.

9. Concise Code: Verbosity doesn't equate to clarity. Clean code is often concise, devoid of unnecessary repetition or fluff.

10. Continuous Refactoring: Clean code isn't a destination but a journey. As you gain new insights or as requirements change, continuously refactor your code to keep it clean.

By internalizing the principles of clean code from the early stages of your career, you don't just set the foundation for professional growth. You pave the way for more efficient, maintainable, and scalable software projects. Remember, in the world of software, the quality of your code often mirrors the quality of the end product.

Get the book on Amazon

Design patterns are like helpful shortcuts for developers. They're not ready-made code but more like problem-solving methods, we can use over and over again. These patterns help solve common design problems without starting from scratch each time.

I've found this book really handy on many projects. Whenever I'm not sure which design pattern to use, I turn to Shvets' clear guidance.

One of my go-to patterns from the book is the State Pattern. It lets an object change how it acts based on its current "state" or situation. Imagine a light switch: it behaves differently when it's "on" versus "off". The State Pattern makes managing such changes in behavior easy and clean.

Whether you're just starting out or have been coding for years, 'Dive into Design Patterns' is a helpful guide. It's all about using smart, tried-and-true methods to make software better.

Get the book

Although familiar with NoSQL databases, DynamoDB's unique features initially seemed a bit unconventional. Concepts such as planning access patterns in advance or the single table approach were intriguing. DeBrie's explanations brought clarity to these concepts.

A key realization, emphasized in the book, is the importance of "thinking in access patterns." While the data structure is crucial, DynamoDB shifts the focus to how the data will be accessed. For a blog, considering how users access posts—whether by author, date, or tags—ensures efficient data retrieval.

DeBrie's insights on the single table design were illuminating. In traditional databases, one might maintain separate tables for related data entities. With DynamoDB, related items can coexist in one table, leveraging proper indexing. It requires an adjusted mindset, but the benefits in terms of flexibility and performance are evident once grasped.

The book also delves deep into best practices for designing tables, offering insights on leveraging DynamoDB's capabilities, especially when operating at scale. DeBrie's expertise aids in understanding the platform's nuances, ensuring optimal utilization while avoiding common pitfalls.

For anyone already familiar with NoSQL but aiming to deepen their knowledge of DynamoDB's unique offerings, 'The DynamoDB Book' is a must-read. It demystifies the advanced concepts and bridges the gap between traditional databases and DynamoDB's innovations.

Get the book

A significant takeaway for me was the importance of simplicity in design, regardless of how intricate the problem or the system might be. This emphasis on simplicity helps ensure the longevity and maintainability of software.

Martin's discussion about managing dependencies was enlightening. It's crucial to ensure that the dependencies within our software don't create complexities that become hard to navigate later on.

One concept that truly resonated with me was the idea of ‘screaming architecture’. It suggests that the structure of the software should clearly reflect its functionality and purpose. This promotes transparency and immediate understanding for anyone diving into the system.

In essence, while "Clean Architecture" aligns with many principles I already knew, it gave depth to those ideas and presented actionable strategies to ensure our software remains robust and easy to evolve over time.

Database pooling is a technique to reduce the opening and closing of connections by maintaining a set of open connections that can be passed among database operations. As a result, the connection overhead is spread across several client requests.

Why use Amazon RDS Proxy?

Opening multiple connections is expensive. Establishing a connection includes DNS lookup to locate the server IP address, handshakes to establish a TCP connection, encryption, validating credentials, and more. In addition, all the established connections will consume some CPU to be maintained.

• Amazon RDS Proxy allows applications to pool and share database connections to improve scalability.

• Enables application resilience by automatically connecting to a standby Database instance when there is a failure while preserving the established connections.

• Additional security as you can enforce AWS Identity and Access Management authentication for databases and securely store the credentials in AWS Secrets Manager.

• Queues or throttles application connections that can’t be served immediately from the pool, thus can keep your application running in a degraded state.

Compare a database instance’s performance before and after implementing RDS Proxy

Now let’s see RDS Proxy in action. We want to see how the instance behaves with and without RDS Proxy. Here are the steps we are going to conduct:

1. Create a db.t3.medium RDS instance. We want at least a t3 so we can leverage Performance Insights for comparisons

2. Create a Cloud9 environment to run load tests for our database instance.

3. Import a test database from https://github.com/datacharmer/test_db. Otherwise you could use your own simple schema since we will just be testing connections.

4. Run mysqlslap commands to emulate database connections using the DNS endpoint for our database instance

5. Create Amazon RDS Proxy

6. Run the same mysqlslap commands using the RDS Proxy endpoint as host

Create Amazon RDS instance

• Navigate to the AWS console and search for RDS

• For the database creation method, select the Standard create option.

• Select MySQL for the Engine options.

• Select Production template

• Select Single DB instance for Availability and durability (in production, you will want to use Multi-AZ)

• On the Settings screen, enter the password and leave the rest of the settings as is.

• For database instance configuration, select Burstable classes and choose at least a db.t3.medium instance size (we need more memory to enable Performance Insights)

• For Storage type, select General Purpose SSD (gp2) and keep the rest of the settings default.

• In terms of Connectivity, keep the default settings, maintaining No public access for the database instance; you can optionally create a new VPC if you like.

• For Database authentication, select Password authentication

• Under Monitoring, make sure Turn Performance Insights is checked

• When done, click the Create Database button to launch your database instance.

Create Cloud9 environment

Create environment

1. Navigate to your AWS console and search for Cloud9.

2. Click on the Create environment button.

3. Provide a name for your environment.

4. Select Additional instance types then choose a m5.xlarge. We are selecting a larger instance because we want more resources for test load scripts. Otherwise, the kernel will kill our processes.

To avoid additional costs, always tear down your AWS resources when done.

5. Proceed with all the remaining steps and create your environment. The creation process may take a few minutes. 6. Give Cloud9 permission to access MySQL. Our Cloud9 environment will need permission to access our database instance. We achieve that by going to the instance’s security group and adding an inbound rule that allows MySQL traffic from our Cloud9 instance as below:

Import test database

AWS Cloud9 EC2 environments come preinstalled with Git. In your Cloud9 environment terminal, run the following commands:

git clone https://github.com/datacharmer/test_db.git && cd test_db

Run the script to create the database schema and load the data. Replace the host with your own database DNS endpoint.

mysql -h database-1.cxyzjeifrqhn.us-east-1.rds.amazonaws.com \ 
-u admin -p \
-t < employees.sql \

You should see output similar to the following:

+-----------------------------+
| INFO                        |
+-----------------------------+
| CREATING DATABASE STRUCTURE |
+-----------------------------+
+------------------------+
| INFO                   |
+------------------------+
| storage engine: InnoDB |
+------------------------+
+---------------------+
| INFO                |
+---------------------+
| LOADING departments |
+---------------------+
...

Test database instance without RDS Proxy

How many connections is too many?

Before we start testing, let's figure out how many connections are too many for our database instance. For example, the maximum number of simultaneous database connections varies by the DB engine type and the memory allocation for the DB instance class. The larger the database instance, the larger the connections it can handle. So there is a formula that you can use to calculate the maximum number of connections. For MySQL, we use:

max_connections = {DBInstanceClassMemory/12582880}

In our case, we are using a db.t3.medium instance with a memory of 4 GiB.

max_connections = 1024*1024*1024*4/12582880 ~= 341

Less than max_connections test

mysqlslap -h database-1.cxyzjeifrqhn.us-east-1.rds.amazonaws.com -u admin -p \
--concurrency=100 \
--create-schema=employees \
--query="SELECT * FROM employees;" \
--verbose
Enter password: 
Benchmark
        Average number of seconds to run all queries: 18.414 seconds
        Minimum number of seconds to run all queries: 18.414 seconds
        Maximum number of seconds to run all queries: 18.414 seconds
        Number of clients running queries: 100
        Average number of queries per client: 1

More than max_connections test

Remember 341 is our max_connections. Let's try with 345.

$ mysqlslap -h database-1.cxyzjeifrqhn.us-east-1.rds.amazonaws.com -u admin -p \
> --concurrency=345 \
> --create-schema=employees \
> --query="SELECT * FROM employees;" \
> --verbose
Enter password: 
mysqlslap: Error when connecting to server: 1040 Too many connections
mysqlslap: Error when connecting to server: 1040 Too many connections
mysqlslap: Error when connecting to server: 1040 Too many connections
mysqlslap: Error when connecting to server: 1040 Too many connections
...
...
mysqlslap: Error when connecting to server: 1040 Too many connections
mysqlslap: Error when connecting to server: 1040 Too many connections
Benchmark
        Average number of seconds to run all queries: 54.603 seconds
        Minimum number of seconds to run all queries: 54.603 seconds
        Maximum number of seconds to run all queries: 54.603 seconds
        Number of clients running queries: 345
        Average number of queries per client: 1

Create RDS Proxy

Click proxies on the left menu of RDS

Our next step is creating the RDS Proxy to pool our connections.

1. Click on the Proxies link on the left menu in the Amazon RDS console.

1. Click Create proxy button and enter name it database-1-proxy or anything.

1. Next, we need a target database instance for the proxy by selecting it from the database dropdown.

1. Supply a secret from AWS Secrets Manager; RDS proxy stores credentials in AWS Secrets Manager, and we will create a new secret.

1. AWS Secrets Manager will open in a new tab. Choose Credentials for Amazon RDS database and enter a username and password. You will need these credentials to connect to your RDS proxy.

Select your database at the bottom and click next.

1. Enter a name for your secret.

1. Now go back to the tab with the RDS Proxy creation form, refresh the secret selection and choose your created secret.

1. Scroll down and click Create proxy. Creation may take up to 20 mins

2. You need to wait until you have both endpoint and target group identifier statuses set to available, as seen below.

1. Copy your proxy endpoint, and we will use that next.

Test database instance with RDS Proxy

More than max_connections test - RDS Proxy

Let's test with the same number of connections that errored out when we didn't use RDS proxy.

mysqlslap -h database-1-proxy.proxy-ctttjeifrqhn.us-east-1.rds.amazonaws.com \
> -u admin -p \
> --concurrency=345 \
> --create-schema=employees \
> --query="SELECT * FROM employees;" \
> --verbose
Enter password: 
Benchmark
        Average number of seconds to run all queries: 65.294 seconds
        Minimum number of seconds to run all queries: 65.294 seconds
        Maximum number of seconds to run all queries: 65.294 seconds
        Number of clients running queries: 345
        Average number of queries per client: 1

Results

You will notice that we didn't get Error when connecting to server: 1040 Too many connections, and we completed our queries (despite being slow).

Counter Metrics

In the chart above, you will notice that without an RDS proxy, we have several db.Users.Aborted_clients. However, when we ran the queries using the proxy, we didn't experience any aborted connections. As a result, we could read more rows, as shown by the db.SQL.Innodb_rows_read metric.

Waits comparison

Waits dimension is an integral part of performance troubleshooting. Waits are reasons why execution threads have to wait. So when you issue queries, various wait stats could prevent your query execution from completing. These can range from CPU or connection waits. In other words, your execution threads will be waiting for resources.

Now, if we look at our chart above, wait/io/socket/sql/client_connection dimension is higher for operations not using an RDS proxy. io/socket/sql/client_connection event occurs when a thread is handling a new connection, and it makes sense that we have many such events when we aren't pooling our database connections with the RDS proxy. So mysqld will be busy creating threads to handle incoming new client connections. Thus, the event io/socket/sql/client_connection slows down the process of servicing new client connection requests since other database connections will be waiting for the thread to be assigned.

Conclusion

RDS proxy can make your applications more scalable, resilient, and secure.

Applications' scalability is improved when they share a pool of database connections. For example, an instance using an RDS proxy can scale better than one, not managing database connections.

Resilience is achieved by a proxy connecting to a standby Database when there is a failure.

Using IAM authentication and retrieving credentials from AWS Secrets Manager improves the security of your database. These credentials can be automatically rotated as AWS Secrets Manager supports that functionality out of the box.

RDS proxy can be handy in serverless applications which support highly variable workloads. For example, the variable workloads may attempt to open a burst of new database connections.

Read more on RDS proxy.

Our final setup will look as below:

Why?

• Replicas benefit read-heavy workloads as they improve performance by minimizing the load on the primary database instance. They make it easy to elastically scale out beyond the capacity constraints of a single DB instance. Before you reach for the next database instance class (scaling up), consider replicas (scaling out). You can use replica endpoints for your ETL processes for the analytics team etc.

• Replicas can be promoted in disaster recovery. If the primary goes offline, a replica can be promoted to become primary. However, replication lag should be considered to avoid data loss. You want to promote a replica that has a near-zero replication lag.

We will follow these steps:

Step 1 - Create a single RDS Instance

Step 2 - Run benchmark queries on our RDS instance using mysqlslap. Mysqlslap is a diagnostic program designed to emulate client load for a MySQL server and to report the timing of each stage. It works as if multiple clients are accessing the server.

Step 3 - Create a read replica(s) from our RDS instance

Step 4 - Use Route53 to distribute read requests across multiple RDS read replicas.

Step 5 - Run the same benchmark queries leveraging Route53 record sets.

Create a single Amazon RDS Instance

• Navigate to the AWS console and search for RDS
• For the database creation method, select the Standard create option.
• Select MySQL for the Engine options.

• Select Production template
• Select Single DB instance for Availability and durability (in production, you will want to use Multi-AZ)

• On the Settings screen, enter the password and leave the rest of the settings as is.
• For instance configuration, select Burstable classes and choose at least a db.t3.medium instance size (we need more memory to enable Performance Insights)
• For Storage type, select General Purpose SSD (gp2) and keep the rest of the settings default.
• In terms of Connectivity, keep the default settings, maintaining No public access for the database instance. You can optionally create a new VPC if you like.
• For Database authentication, select Password authentication
• Under Monitoring, make sure Turn Performance Insights is checked
• When done, click the Create Database button to launch your database instance.

Benchmark queries using mysqlslap

We can run the benchmark queries in an Amazon EC2 instance or AWS Cloud9 environment. Our database instance is not accessible publicly, so we need to access it within our VPC. I’m going to use the Cloud9 environment. AWS Cloud9 is a cloud-based integrated development environment (IDE) that allows you to write, run and debug code via the browser.

Cloud9 Setup

Create environment

1. Navigate to your AWS console and search for Cloud9.
2. Click on the Create environment button.
3. Provide a name for your environment.

1. Click next, and you should be ok with the default settings.

1. Proceed with all the remaining steps and create your environment. The creation process may take a few minutes.

2. Give Cloud9 permission to access MySQL. Our Cloud9 environment will need permission to access our database instance. We achieve that by going to the instance’s security group and adding an inbound rule that allows MySQL traffic from our Cloud9 instance as below:

Clone sample database repository

Next, we must pull in a sample database to test our database instances. The sample database can be found at https://github.com/datacharmer/test_db, and you can also read more about it at https://dev.mysql.com/doc/employee/en/.

AWS Cloud9 EC2 environments come preinstalled with Git. In your Cloud9 environment terminal, run the following commands:

git clone https://github.com/datacharmer/test_db.git && cd test_db

Run the script to create the database schema and load the data. Replace the host with your own database DNS endpoint.

mysql -h database-1.ctlblrdarm7c.us-east-1.rds.amazonaws.com \ 
-u admin -p \
-t < employees.sql \

If you get an error similar to ERROR 2002 (HY000): Can't connect to MySQL server on … go back and make sure your Cloud9 environment security group has inbound permission to your database instance security group.

Otherwise, you should see output similar to below:

mysql -h database-1.ctlblrdarm7c.us-east-1.rds.amazonaws.com \ 
-u admin -p -t < employees.sql 
Enter password: 
+-----------------------------+
| INFO                        |
+-----------------------------+
| CREATING DATABASE STRUCTURE |
+-----------------------------+
+------------------------+
| INFO                   |
+------------------------+
| storage engine: InnoDB |
+------------------------+
+---------------------+
| INFO                |
+---------------------+
| LOADING departments |
+---------------------+
+-------------------+
| INFO              |
+-------------------+
| LOADING employees |
+-------------------+
+------------------+
| INFO             |
+------------------+
| LOADING dept_emp |
+------------------+
+----------------------+
| INFO                 |
+----------------------+
| LOADING dept_manager |
+----------------------+
+----------------+
| INFO           |
+----------------+
| LOADING titles |
+----------------+
+------------------+
| INFO             |
+------------------+
| LOADING salaries |
+------------------+
+---------------------+
| data_load_time_diff |
+---------------------+
| 00:00:39            |
+---------------------+

Run mysqlslap on primary instance

We are going to use the following mysqlslap command to emulate load on our primary database instance.

mysqlslap -h database-1.ctlblrdarm7c.us-east-1.rds.amazonaws.com -u admin -p \
--concurrency=10 \
--iterations=15 \
--create-schema=employees \
--query="SELECT * FROM employees;SELECT * FROM titles;SELECT * FROM dept_emp;SELECT * FROM dept_manager;SELECT * FROM departments;" --delimiter=";" \
--verbose

Let’s explain some of the mysqlslap options.

Database load sliced by Waits

Database load sliced by SQL queries

Create read replicas

Now let’s create our read replicas.

1. Navigate to your AWS RDS console
2. Select your primary database instance, and from the Actions drop-down click Create read replica
3. Add a title for your read replica as below. Keep the other defaults and create the replica
4. Repeat the step above and create a second replica named database-1-replica-2. When you are done, you will have a primary instance and two replicas as below.

Each database instance (including replicas) will have a unique DNS connection string. However, if we were in an application, we wouldn’t want to manage connecting to the different strings. Also, if we add or remove a replica, we will need to update the connection strings in our application. A good practice would be to use Route 53 to distribute read requests across multiple Amazon RDS instances. Therefore, we will tackle that next.

Use Route 53 to load balance Amazon RDS read replicas

We will use Amazon Route 53 weighted record sets to distribute requests across our read replicas. Weighted routing lets us associate multiple resources with a single domain name or subdomain. It allows us to choose how much traffic is routed to each resource.

Create Route 53 hosted zone

1. Search for Route 53 in the AWS console.
2. Navigate to Hosted Zones and select Create hosted zone.
3. Enter any Domain name and select a Private hosted zone for the type. Our database instances are only accessible within the VPC. Any domain name works here because we are creating a private hosted zone.
4. Select the same VPC that your database instances are launched in
5. Click Create hosted zone

Create hosted zone CNAME records

Now that we have our hosted zone created, the next task is the creation of our zone CNAME records. Each database instance DNS endpoint will map to a specific CNAME record. We will still accept reads on our primary database instance, so we will add a record for the primary instance.

1. Create the primary database instance CNAME record by clicking Create record. Fill in the form with the following fields:

Record name - db

Record type - CNAME

Value - YOUR_PRIMARY_DATABASE_DNS_ENDPOINT

TTL - 60

Routing policy - Weighted

Weight - 100

Record ID - ANY_UNIQUE_VALUE_WITHIN_RECORDS

1. Repeat the above for the two replicas. Make sure to use DNS endpoints for the replicas in the Value field and add unique Record IDs. When done, your record list should look as below.

We should now be able to connect to MySQL using the Record name. For example, mysql -h db.domain.local -u admin -p. We should only use this record name for reads because database writes will fail when we get routed to replicas (by default, replicas are read-only).

Run mysqlslap on primary instance and read replicas

We will use the same mysqlslap command as before. The only change is our database host which is db.domain.local this time.

$ mysqlslap -h db.domain.local -u admin -p \
--concurrency=10 \
--iterations=15 \
--create-schema=employees \
--query="SELECT * FROM employees;SELECT * FROM titles;SELECT * FROM dept_emp;SELECT * FROM dept_manager;SELECT * FROM departments;" --delimiter=";" \
--verbose

After the command is done, give Performance Insights a little time and you should see a difference in database load.

Database load on the primary instance before and after adding read replicas.

Database load sliced by SQL queries

Database load sliced by Waits

Conclusion

As you can see, for the same queries and clients, the load is reduced on our primary database instance because we are offloading some of the reads to our read replicas. Our primary database instance can take care of writes without being swamped with reads. You can even go further and only use the primary database for writes only. You can achieve that by deleting the primary database CNAME record from Route 53 or by setting the weight value to a value of zero. We could also have created a CNAME record for writes and routed our write traffic via Route 53.

In object oriented programming, the command pattern is a behavioral design pattern in which an object is used to encapsulate all information needed to perform an action or trigger an event at a later time.

The information that is encapsulated in a commanding object includes the method name, the object that owns the method, and values for the method parameters. So, in other words, the commanding entity will have enough information to take action immediately or later.

Example Usage

Let’s take, for example, the method below that sends a message to other users in a conversation in a chat application.

/**
 * Sends a message
 *
 * @param Conversation $conversation
 * @param array $data contains message body and any other relevant information
 * @param User $sender
 *
 */
public function send($conversation, $data, $sender)
{
    //Step 1. Send a message by adding the $sender message to the $conversation

    //Step 2. Maybe we are sending a huge file and would rather que that for processing and let the other part of an incoming file

    //Step 3. Send a push notification to each user in the $conversation

    //Step 4. Send an email notification to each user in the $conversation

    //Step 5. Store notifications in the database for each user in the $conversation

    //Step 6. Tell the sender that their message has been sent
}

We could start coding our application and implement all the above steps in the send() function. However, by doing that, we are well on our way to writing code that is not efficient and may not be easy to maintain. For instance, the is no reason why Step 4 above should wait for items like sending notifications. So what if we can move all those steps somewhere else? That’s where the Commanding pattern can help us.

So let’s attempt to use the commanding pattern instead.

<?php

namespace Musonza\Chat;

use Musonza\Chat\Commanding\CommandHandler;
use Musonza\Chat\Messages\SendMessageCommand;

class Chat {

    public function send($conversation, $data, $sender)
    {
        //$this->message refers to Message model
        $command = new SendMessageCommand($conversation, $this->message, $data, $sender);

        //CommandHandler
        $this->commandHandler->handle($command);
    }

}

In the function above, we encapsulate all information needed to send a message in the SendMessageCommand object. Let’s take a look at the SendMessageCommand class:

SendMessageCommand

<?php

namespace Musonza\Chat\Messages;

use Musonza\Chat\Commanding\Command;

class SendMessageCommand implements Command
{
    public $sender;
    public $body;
    public $conversation;

    public function __construct($conversation, $message, $body, $senderId)
    {
        $this->conversation = $conversation;
        $this->message      = $message;
        $this->data         = $data;
        $this->sender       = $sender;
    }

    public function execute()
    {
        $this->message->send($this->conversation, $this->data, $this->sender);
    }

}

As you can see from the command constructor, we are setting the values for parameters that we need to perform our action of sending a message. Below is the interface that all our commands will implement. That way, we are sure each command has an execute() method.

<?php

namespace Musonza\Chat\Commanding;

/** The Command interface */
interface Command {
   public function execute();
}

CommandHandler

<?php

namespace Musonza\Chat\Commanding;

/** The Invoker class */
class CommandHandler {
   public function handle($command) {
      //here we can store the command or do anything else
      $command.execute();
   }
}

Summary

These are the steps we took in sending our chat message using the commanding pattern.

1. We made a call to the send() function with all the required information to send a message

2. We passed on that information to the Commanding object SendMessageCommand.

   • Now the commanding object knows the information needed to send a new message

   • We can push this commanding object to a que for later execution or we can execute right away

3. We then made a call to CommandHandler method handle() which would cause the commanding object to call it’s execution method.

In our case, when the execution method is called, we send the message by making this call:

$this->message->send($this->conversation, $this->data, $this->sender);

Note: The code presented here is shortened. Feel free to checkout a Laravel Package I created that implements the commanding pattern.

Security

While AWS has created a secure cloud infrastructure, it is essential to note that security is not Amazon's sole responsibility. Instead, Amazon uses the Shared Responsibility Model, which states that AWS manages the security of the cloud, and security in the cloud is the customer's responsibility.

IAM Roles

When you have EC2 instances and your application needs to access other AWS services (like SNS, S3, etc.), there are two ways to grant permission to your EC2 instances. You can either manually assign credentials to the instance or use IAM Roles. The best practice is always to use the latter. Manually assigned credentials need to be rotated frequently, which can be messy when you have a lot of instances. You also risk getting the credentials extracted; thus, your applications are more vulnerable.

So, unless you connect from outside AWS (like your computer), you always need to use IAM Roles to grant permission.

Storage and Content delivery

Simple Storage Service

Data storage for your applications can be costly. There are a few factors to consider when storing data on AWS. These factors include; how often is the data accessed? How fast should be the data retrieval process? Can the data be reproduced easily? How long should the data be available?

A popular storage service on AWS is Simple Storage Service (S3). S3 can serve objects through a CDN to Cloudfront, can serve HTML files with Route 53, and allows managed access and versioning of objects. With S3, you can set lifecycle policies for your objects. For instance, you can specify a policy that deletes or archives your objects to Amazon Glacier after a certain period.

So if your application needs frequent access to the data, S3 is a service of your choice. You can further reduce your cost using S3 Reduced Redundancy Storage(RRS) instead of S3 standard storage. However, it would be best to use only RRS for easily reproducible data (thumbnails, etc.) or for that you can afford to lose.

If you have data that your application doesn't need to frequently access and when accessed, the retrieval time is not of importance; Amazon Glacier would be suitable. Amazon Glacier offers an archival storage type. Checking in and checking out items in Amazon Glacier can take several hours, hence the use for archival purposes only.

Databases

Relational databases

If your application needs to use a relational database, AWS offers RDS, a fully managed database for relational databases. Being a fully managed database means Amazon will handle the underlying software updates and patches. RDS supports the following databases: PostgreSQL, MySQL, Oracle, SQL, and Aurora. Aurora is just a more efficient fork of MySQL, and Amazon recommends using Aurora in production environments.

Another recommendation in production is RDS Multi-AZ deployment. When a Multi-AZ DB Instance is provisioned, AWS automatically creates a primary DB instance and synchronously replicates the data to a standby instance in a different Availability Zone. In the event of a failure on the primary DB instance infrastructure, AWS will perform an automatic failover to the standby database. There will be minimal or no downtime at all experienced by your application. Multi-AZ deployment is also important when AWS updates and patches the underlying database software. AWS will update primary and standby database asynchronously meaning your application is not affected.

NoSQL

Update: AWS now has Amazon DocumentDB service. Amazon DocumentDB is a scalable, highly durable, and fully managed database service for operating mission-critical MongoDB workloads.

A popular NoSQL database in the community is MongoDB. Usually, if they want to keep all the services on AWS, developers set up a MongoDB database on an EC2 instance. That means as a developer, you are responsible for any software updates to your database and ensuring the database is secure from the outside world. You also have to make sure the database is scalable. This sounds like a lot of work and responsibility already. An alternative would be DynamoDB, a fully managed NoSQL database service provided by AWS. DynamoDB is fully distributed and auto-scales which makes it fault-tolerant. AWS also manages the provisioning of all underlying hardware. You can specify the required throughput capacity as a developer, and DynamoDB will handle the rest.

Caching

You can further improve the performance of your application by implementing caching. For instance, you can cache database query results to avoid constantly hitting the database. You can also cache web sessions and any dynamically generated content. Amazon ElastiCache, a fully managed in-memory cache engine can be used with Redis or Memcached to achieve this.

Deployment Services

Elastic Beanstalk

If you want to get up and running quickly or don’t have the technical knowledge for building application environments, Elastic Beanstalk can be a deployment service of your choice. With Elastic Beanstalk, you can quickly deploy an entire application environment automatically. The service integrates with other AWS services, including Elastic Load Balancer, Auto Scaling, and EC2.

Amazon CloudFormation

CloudFormation offers developers an easy way to create and manage a collection of related AWS resources. It is essentially Infrastructure as code as CloudFormation templates are merely JSON files. This can be useful if you are to scale your application; you can use a CloudFormation template to build EC2 instances that belong to an Elastic Load Balancer. You can also use the templates in disaster recovery, reducing the time required to spin up a new environment. This was just an overview of the services that Amazon AWS has to offer. There are many essential services we didn’t cover, including Networking (Virtual Private Cloud, DNS etc.)

We believe API design should follow an open/closed principle: "software entities should be open for extension but closed for modification." Applying this principle will help your API evolve as the industry changes. Here are some of the best practices and pitfalls we've discovered to help you build a great API yourself.

Use nouns instead of verbs (RESTful)

Verb resources (bad practice)

Developers may be tempted to be as descriptive as possible when structuring API resources. Take, for instance, the table above. Depending on your application's size, using verbs will result in a massive list of URLs. It's hard to keep track of such a list as a developer, but it's even harder for your API consumers as they will need to dig into the documentation for every task. An API should be designed so that users can guess or experiment successfully with the URLs.

Noun resources

Above is an example of how we can structure our resources better and make the API easier to use. For example, we only need two resources, /photos and /photos/:id, in conjunction with four HTTP verbs (GET, POST, PUT, DELETE) to be able to perform the operations in our previous colossal list.

Use Transformers for data presentation

Another bad practice is querying the database and dumping everything as an API response. Instead, I recommend using transformers over dumping the database data to the world.

Transformers give you control over what data you output to your API consumers. With transformers, we are confident that sensitive information is not displayed by mistake. Not only is it a security measure, but transformers will also help prevent API consumers' code from breaking when your database schema changes.

While you can build your transformers easily, I recommend Fractal if you are in the PHP world. With transformers, you are afforded the benefit of serializing your data in a specific way and embedding related resources within each other.

Error handling

Use HTTP status codes

While there are more than a dozen status codes, we only need ten codes for REST. Your API consumers will know how to respond to the errors since the status codes are a standard. It would be best if you didn’t make your consumers rely on the error messages to determine how to proceed from an error. Below are the ten status codes you need to use:

So don’t just return HTTP status code 200 for every request.

Error Codes and Messages

Apart from returning the correct HTTP status codes, it also helps to return custom error codes and verbose error messages for your API. These will go a long way if you have to offer support to your API consumers. Knowing the error code and message will make it easy to pinpoint the issue in your code.

Documentation

Are you happy to use an API without documentation? I’m hoping you answered “no.” So while documentation may be painful, you have to do it. Good resource documentation should at least include the following:

• Resource URL (https://api.example.com/photo/:id)
• Resource Information (Description, Response formats including - JSON and XML, Requires authentication?)
• Parameters (specify required/optional) and give examples
• Example request
• Example result

Including usage examples and tutorials will be a plus.

Including usage examples and tutorials will be a plus.

Authentication

Another critical aspect of API design is authenticating and protecting our applications. Below are some strategies you can use to authenticate clients using your API.

HTTP Basic authentication

This is a standard HTTP authentication mechanism whereby clients supply a username and password BASE64 encoded in the “Authorization” header.

Oauth2

Ouath2 comprise of the following two:

1. user/client credentials to use in getting an access token
2. Use the access token to access the API resources

However, since this involves transmitting access tokens and authentication credentials, Oauth2 is only secure if used with TLS/SSL to encrypt data during transmission. You also need to consider storing the client credentials on clients like mobile applications. There is a debate about this, as keeping the credentials for Oauth2 on a mobile or desktop app isn’t the most secure thing to do. If, for instance, our client credentials get compromised, then the Oauth2 provider has to revoke the entire application since we would have that information hard coded.

We have only scratched the surface here. There are a lot of techniques that we could talk about to make our APIs scalable and secure.

Decoupling

Test-Driven Development forces you to write tests before implementation. This, in turn, causes you to be concise since you must define your input and expectations before writing any code. When your code is derived from tests, you inevitably separate the different pieces as much as possible. Decoupled code is easily maintainable.

Up to date documentation

While it seems like a good idea to only rely on a living document, as time goes on and developers add more features or fix bugs, that document usually becomes neglected. However, enforcing TDD means developers must add new tests for new features or write regression tests for bugs. As a result, tests can act as documentation that must stay updated.

Refactor with confidence

Have you ever been unable to refactor code because of the fear that something will break? I have seen code that cries out loud for refactoring (including some code I have written in the past). I have sat there and watched classes grow into monster objects with duplicates because refactoring would be like opening a pandora's box. With tests, you can refactor code confidently because you are always going to run tests after each refactor, and if the test fails, you can immediately revert it.

Minimize deployment errors

If you are often afraid to deploy to production because the result is unknown, TDD can help you allay those fears. Writing code without bugs doesn't mean that it works. Adding new features to a project will be a manageable task. I have worked on projects where I have come back a few months or years later to add features. Without written tests, I always had to keep my fingers crossed that nothing broke. Well-written tests will eliminate this headache as the tests will provide constant feedback that each component is still working.