Piotr Mionskowski

First steps with AWS Bedrock

Sun, 17 Dec 2023 00:00:00 GMT

AI is taking over the world. At Bright Inventions, we’ve already helped several clients with generative AI.
In this blog post, we’ll see how to use aws-cdk to create a simple API that responds to prompts.

Request Bedrock model access

If you haven’t used Bedrock before, the first step is to request model access.

You can do so in AWS Console > Bedrock Model > access page:

For Claude and Claude Instant models describe your use case briefly.

Define your AWS Lambda function with aws-cdk

Declaring an AWS Lambda function with aws-cdk is straightforward. Our function needs to invoke Bedrock models, hence appropriate IAM permissions are necessary. For simplicity, we’ll use AWS Lambda Function URLs with authentication type NONE. You should use that only for evaluation purposes.

export class BrightBedrockSimpleStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const lambdaPrompt = new NodejsFunction(this, 'lambda', {
      architecture: Architecture.ARM_64,
      timeout: Duration.seconds(30),
      entry: path.join(process.cwd(), 'lib', 'bedrock-client', 'simple-api.lambda.ts'),
    })

    lambdaPrompt.addToRolePolicy(new PolicyStatement({
      actions: ['bedrock:InvokeModel'],
      resources: ['*']
    }))

    const functionUrl = lambdaPrompt.addFunctionUrl({
      authType: FunctionUrlAuthType.NONE
    });

    new CfnOutput(this, 'function-url', {
      value: functionUrl.url
    });
  }
}

Invoking Bedrock amazon.titan-text-express-v1

Bedrock provides multiple models. The models differ not only in terms of their capabilities but also in their API. For starters, let’s use Titan Text Express.

export const handler: APIGatewayProxyHandlerV2 = async (event, context) => {
  const body = event.body!

  const response = await bedrock.send(new InvokeModelCommand({
    body: JSON.stringify({
      inputText: body,
    }),
    contentType: 'application/json',
    modelId: "amazon.titan-text-express-v1"
  }));

  const modelResponseJson = response.body.transformToString();

  return {
    statusCode: response.$metadata.httpStatusCode ?? 500,
    headers: {
      'Content-Type': response.contentType
    },
    body: modelResponseJson
  }
}

After we deploy the code, we can invoke our function and ask basic questions, e.g.

curl -s -X POST --location "https://${YOUR_LAMBDA_ID}.lambda-url.eu-central-1.on.aws" \
    -H "Content-Type: text/plain" \
    -d 'Which country has the highest GDP?' | jq
{
  "inputTextTokenCount": 7,
  "results": [
    {
      "tokenCount": 34,
      "outputText": "\nThe country that has the highest GDP is the United States. Its total GDP is $23.07 trillion in terms of purchasing power parity (PPP).",
      "completionReason": "FINISH"
    }
  ]
}

Titan Text Express configuration

We can control and tweak some the aspects of how the model responds to our prompts. For example, for Titan Text Express we can configure:

temperature: Float value to control randomness in the response (0 to 1, default 0). Lower values decrease randomness.
topP: Float value to control the diversity of options (0 to 1, default 1). Lower values ignore less probable options.
maxTokenCount: Integer specifying the maximum number of tokens in the generated response (0 to 8,000, default 512).
stopSequences: Array of strings indicating where the model should stop generating text. Use the pipe character (|) to separate different sequences (up to 20 characters).

Let’s modify our lambda to allow controlling the parameters.

const response = await bedrock.send(new InvokeModelCommand({
  body: JSON.stringify({
    inputText: body,
    textGenerationConfig: {
      temperature: parseFloat(event.queryStringParameters?.temperature ?? '') || undefined,
      topP: parseFloat(event.queryStringParameters?.topP ?? '') || undefined,
      maxTokenCount: parseInt(event.queryStringParameters?.maxTokenCount ?? '') || undefined,
    }
  }),
  contentType: 'application/json',
  modelId: "amazon.titan-text-express-v1"
}));

After deploying, we can control the model parameters through query string parameters:

curl -X POST --location "https://g66m7el4jcrtwarntzrcfps2tq0yduar.lambda-url.eu-central-1.on.aws?temperature=0.9" \
    -H "Content-Type: text/plain" \
    -d 'What is the country that has the most freedom of speech in the world?'
    
{
  "inputTextTokenCount": 15,
  "results": [
    {
      "tokenCount": 28,
      "outputText": "\n\"The United States has the most freedom of speech in the world, according to the 2022 Freedom House Index.\" ",
      "completionReason": "FINISH"
    }
  ]
}

The results will be more elaborate if we change topP.

curl -X POST --location "https://g66m7el4jcrtwarntzrcfps2tq0yduar.lambda-url.eu-central-1.on.aws?temperature=0.9&topP=0.1" \
    -H "Content-Type: text/plain" \
    -d 'What is the country that has the most freedom of speech in the world?'
    
{
  "inputTextTokenCount": 15,
  "results": [
    {
      "tokenCount": 51,
      "outputText": "\n\"The United States is considered the country that has the most freedom of speech in the world. This freedom is protected by the First Amendment of the U.S. Constitution, which guarantees the right to freedom of expression, assembly, and religion.\" ",
      "completionReason": "FINISH"
    }
  ]
}

Summary

As you see, it is straightforward to get started with AWS Bedrock. The full example of this blog post is available in GitHub repo.

Migrate EC2-Classic RDS to a VPC - Step 2 - DNS

Sun, 01 May 2022 00:00:00 GMT

In the plan blog post I sketched a plan to migrate an EC2-Classic RDS database to a VPC. Now we can perform the 2nd step of the plan. This is a simple step. Albeit, we have to understand what are the consequences. In your case, this approach might need adjustments!

The RDS endpoint DNS

When you create an RDS database instance or RDS database cluster AWS provides you with endpoint addresses. The addresses can be used to connect to the database. You cannot change the endpoint address of the RDS database. It is something generated by AWS that you have no influence over.

As planned we want to update the DNS entry when migrating to VPC based RDS instance. For this to work, we need a custom DNS entry. One that we can control.

Private hosted zone

Often, when deploying services inside a VPC it makes sense to use a private DNS hosted zone. Such an approach allows us to define e.g. internal DNS space which we can freely use. With AWS-CDK defining such hosted zone is simple:

class NetworkStack extends Stack {
  readonly privateHostedZone: IPrivateHostedZone
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);
    
    const vpc = new Vpc(this, 'vpc', {
      ...
    })

    this.privateHostedZone = new PrivateHostedZone(this, 'internal', {
      vpc: vpc,
      zoneName: 'internal',
      comment: 'private DNS space for registering services'
    });
  }
}

Custom DNS record for database endpoint

Now that we have a private hosted zone we can define a custom DNS record for the database. The record should be used to connect to the database from our service(s). This way once we change the record the application will establish connections to a new database running inside VPC.

new CnameRecord(this, 'database cname record', {
  domainName: 'current-classical-database-endpoint',
  zone: this.privateHostedZone,
  recordName: 'database'
})

With the above configuration deployed all that is left to do is to update our service configuration. The database hostname that we set should now look like database.internal. Please note that we are still connected to the non-VPC, classical RDS instance.

Important considerations

The approach that we have selected is simple. However, it is not without downsides. The main issue is that the DNS update will not immediately cause all services to connect to the new database. In many cases this is fine. Moreover, we need to make sure that our database clients do not cache DNS entries for too long.

If you need the connections to be moved to a new database instance in a more controllable fashion it is still possible. A deployment with an update of the database configuration of services might be enough. If such a deployment is not an option we can introduce a network proxy e.g. using a network load balancer, ProxySQL or HAProxy.

Migrate EC2-Classic RDS to a VPC - Step 1 - Replication

Sun, 30 Jan 2022 00:00:00 GMT

In the last blog post I sketched a plan to migrate an EC2-Classic RDS database to a VPC. It is time to dive into the details of setting up replication. There are great blog posts with an in-depth explanation of the SQL and PostgreSQL related aspects . We are going to focus on the automation of the replication setup using AWS-CDK.

The source EC2-Classic RDS database parameters

In the case of our client, the EC2-Classic RDS PostgreSQL database was deployed with CloudFormation. With AWS-CDK it is possible to import an existing CloudFormation template and thus gradually switch to a more modern approach. Once imported it is easy enough to update the parameters through ParameterGroup:

const included: CfnInclude = loadCloudFormationTemplate();

const cfnDatabase: CfnDBInstance = included.getResource("Database") as CfnDBInstance;

const parameterGroup = new ParameterGroup(this, 'Replication Source Parameters', {
  engine: DatabaseInstanceEngine.postgres({
    version: PostgresEngineVersion.VER_11
  }),
  parameters: {
    max_replication_slots: '10',
    max_wal_senders: '10',
    max_worker_processes: '10',
    track_commit_timestamp: '1',
    wal_level: 'logical',
  }
});

cfnDatabase.dbParameterGroupName = parameterGroup.bindToInstance({}).parameterGroupName

If the database has a custom parameter group we need to modify its properties instead of creating a new one. Changing the above database parameters sometimes requires a reboot. It is thus important to schedule it ahead of time in the low traffic hours.

Create a replication user with AWS-CDK

It is best to create a dedicated replication user at a database level. Running SQL through CloudFormation can be cumbersome. Thankfully we have developed the cloudformation-sql-run AWS-CDK construct to make this easier.

const replicationUserPassword = new secretmanager.Secret(this, 'Replication User Password', {
  generateSecretString: {
    excludeCharacters: DEFAULT_PASSWORD_EXCLUDE_CHARS
  }
})

const sourceDatabaseConnection = SqlRunConnection.fromDriverTypeHostPort({
  password: SqlSecret.fromSecretsManager(sourceDatabasePassword),
  username: cfnDatabase.masterUsername!,
  driverType: 'postgresql',
  database: cfnDatabase.dbName!,
  host: cfnDatabase.attrEndpointAddress!,
  port: cfnDatabase.attrEndpointPort! as any as number
});

const createReplicationUser = new SqlRun(this, 'Create Replication User', {
  vpc: vpc,
  connection: sourceDatabaseConnection,
  up: {
    run: [{
      sql: `CREATE USER pgrepuser WITH password '${replicationUserPassword.secretValue}'`
    }, {
      sql: `GRANT rds_replication TO pgrepuser`,
    }, {
      sql: `GRANT SELECT ON ALL TABLES IN SCHEMA public TO pgrepuser`,
    }],
  },
  down: {
    run: [{
      sql: `DROP USER IF EXISTS pgrepuser`
    }]
  }
});

Setup replication slots at source RDS database

We create a dedicated SQL user for replication purposes. Now it is time to prepare for the replication itself. Thankfully this is easy and all we need is 2 additional SQL commands

const createPublication = new SqlRun(this, 'Create Publication', {
  vpc: vpc,
  connection: sourceDatabaseConnection,
  up: {
    run: [{
      sql: `CREATE PUBLICATION pglogical_rep01 FOR ALL TABLES`
    }, {
      sql: `SELECT pg_create_logical_replication_slot('pglogical_rep01', 'pgoutput')`
    },],
  },
  down: {
    run: [{
      sql: `DROP PUBLICATION IF EXISTS pglogical_rep01`
    }]
  }
});

Create snapshot at the original database

To speed up replication we’ll use take a snapshot and restore approach. First, we use a snapshot is to create a new VPC based RDS database. Then we set up a logical replication using the above preparation. Unfortunately at the moment taking a snapshot of a database with AWS-CDK requires a custom construct. I will create another blog post describing the details of such construct. However, for the time being, it is enough for you to know that creating and using it is relatively easy:

const cfnDatabase: CfnDBInstance = included.getResource("Database") as CfnDBInstance;

const sourceDataseSnapshot = new DatabaseSnapshot(this, 'Source Database Snapshot', {
  dbInstanceIdentifier: cfnDatabase.dbInstanceIdentifier
})

The snapshot name is going to be available at sourceDataseSnapshot.dbInstanceSnapshotIdentifier

Create a new database from snapshot

It is time to finally create a new RDS database in the target VPC. We create the database from the snapshot created above. The database will use the same database parameter group. It will not receive any production traffic yet. Notice that we create a database cluster instead of a database instance. It doesn’t matter though. The procedure is the same for the database instance.

const vpc: IVpc = props.vpc
const targetDatabaseCluster = new DatabaseClusterFromSnapshot(this, 'Database', {
  instanceProps: {
    vpc: vpc,
  },
  engine: DatabaseClusterEngine.auroraPostgres({
    version: AuroraPostgresEngineVersion.VER_11_4
  }),
  snapshotIdentifier: databaseSnapshot.dbInstanceSnapshotIdentifier,
  parameterGroup: parameterGroup
})

Finding out where to start replication from

Once the database cluster restored from a snapshot is ready we have to start replication. First we create a subscription. Notice that we’re using references to source RDS database.

const createSubscription = new SqlRun(this, 'Create Subscription', {
  vpc: vpc,
  connection: SqlRunConnection.fromDatabaseCluster(targetDatabaseCluster),
  up: {
    run: [{
      sql: `CREATE SUBSCRIPTION pglogical_sub01 
CONNECTION 'host=${cfnDatabase.attrEndpointAddress!} port=5432 dbname=${cfnDatabase.dbName!} 
user=pgrepuser password=${replicationUserPassword.secretValue}' PUBLICATION pglogical_rep01
WITH (
  copy_data = false,
  create_slot = false,
  enabled = false,
  connect = true,
  slot_name = 'pglogical_rep01'
)`
    }, {
      sql: `SELECT 'pg_'||oid::text AS "external_id"
FROM pg_subscription 
WHERE subname = 'pglogical_sub01'`
    }],
  },
  down: {
    run: [{
      sql: `DROP SUBSCRIPTION IF EXISTS pglogical_sub01`
    }]
  }
});

Now that we have a subscription ready to synchronize data. However, we need to advance the replication into the correct position. To do that we need to find the latest log file positions. The RDS service provides API for that. For now, we’ll assume we have a LatestLogPosition AWS-CDK construct that reads the position for us. In the next blog post, I’ll describe how to create such a construct.

const externalId = createSubscription.getStatementResult(0, 'external_id')

const latestLogFile = new LatestLogPosition(this, 'Replication Position', {
  cluster: targetDatabaseCluster
})


const advanceAndEnableSubscription = new SqlRun(this, 'Enable Subscription', {
  vpc: vpc,
  connection: SqlRunConnection.fromDatabaseCluster(targetDatabaseCluster),
  up: {
    run: [{
      sql: `SELECT pg_replication_origin_advance(:externalId, :logFilePosition)`,
      parameters: {
        externalId: externalId,
        logFilePosition: latestLogFile.logFilePosition
      }
    }, {
      sql: `ALTER SUBSCRIPTION pglogical_sub01 ENABLE`
    }]
  },
  down: {
    run: [{
      sql: 'ALTER SUBSCRIPTION pglogical_sub01 DISABLE'
    }]
  }
})

In the last SQL statement, we have enabled the subscription. Now the VPC based RDS database is synchronized with EC2-Classic instance! For the above to work we need to make sure the source database allows establishing connections from VPC-based database on a network level.

Summary

Automating the above process is cumbersome. However, it gives us a solid way to test the approach in a pre-production environment. This way we can find any loopholes in our approach. We can also better estimate how long the procedure is going to take on production. We have also not caused any downtime or degradation of performance to the source database. This is very important since it is still used by our clients!

Migrate EC2-Classic RDS to a VPC - Planning

Fri, 21 Jan 2022 00:00:00 GMT

By end of August AWS will shut down EC2-Classic networking . All resources using EC2-Classic networking will stop working. Yes, this includes RDS databases.

One of our clients was running a PostgreSQL RDS instance in EC2-Classic. We’ve notified the client about the necessary migration right after AWS published the news. Since the database was used for mission-critical features it was vital to plan the migration carefully.

The plan

There are several risks at play when migrating data. They can be divided into 2 major categories:

a) data loss

b) service outage

Preparing a thorough migration plan helps to reduce those risks. Losing data was not an option. We could also afford only limited degradation of service. With a database size of several hundreds of GB, any operation isn’t instant. It will take a couple of hours just to copy the data. The client agreed that it would be ok if the service showed stale data for a couple of minutes.

With the constraint defined we’ve come up with the following plan.

Step 1: Replicate database

As I mentioned it will take several hours to just copy the data. This rules out a simple take snapshot & restore approach. Fortunately, databases have solved this problem a long time ago with replication. The first step is thus to create a fresh new RDS database inside VPC. We can use a snapshot of the source database to speed up initial replication. Please note that at this point the fresh database isn’t used by any production service yet! Once the data wills start flowing from the EC2-Classic RDS instance to VPC based one we can safely prepare for the next step.

Step 2: Setup a custom database host DNS record

We need to have a way to instruct all clients connecting to the database to switch to a new database. One way to achieve that is to use DNS. During the migration, we’ll update the DNS record. This method isn’t ideal. Existing connections will not be automatically served. However, in the case of our clients, it was perfectly ok to request rolling reboots of database client service.

Step 3: Migration!

Once we have a fresh, up-to-date database instance ready inside VPC we can finally switch to the new database. Simply swapping the DNS record would put us at risk of violating data integrity. To mitigate that we’ll put the source database into read-only mode right before swapping the DNS record. Such a change ensure that once the DNS switch takes place, any writes happen only at the VPC based database!

Conclusion

Migrating production services is challenging and risky. However, with the right plan, the risk can be mitigated to an acceptable level. In my opinion, automating most if not the whole process reduces the risk further. This is why we’ve implemented the migration from EC2-Classic to VPC using AWS-CDK.

In the following posts, I’ll share more details on the automation that we’ve implemented.

Step 1: Replicate database

5 Top Programming Books For Junior and Mid-level Developers

Fri, 31 Dec 2021 00:00:00 GMT

I encourage everyone to read books. There are so many titles worth reading that entire lists have been created. Depending on the stage of your career you’ll need books covering various topics. Here is a list I recommend every junior and mid-level software engineer should read.

Head First Design Patterns by Eric Freeman, Elisabeth Robson

Design patterns surround us everywhere. They are prevalent in every framework and most libraries we use. The “Head First Design Patterns” presents the most common design patterns in a succinct and easy to remember fashion. Before you’ll finish the book, you’ll already better understand the design choices behind libraries and frameworks that you use! The form of writing that the book uses is especially enjoyable. You’ll not be bored by this one.

Clean Architecture by Robert C. Martin

Every book by “Uncle Bob” is worth reading. I’ve picked the “Clean Architecture” for the list as it provides a base and set of pillars with which you can create software services that are easier to reason about. The book describes many important rules. At its core, a high-level approach to application architecture is summarized in the author’s blog post . Make no mistake though, you’ll gain much more from reading the book end-to-end!

Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation by Jez Humbe and David Farley

The book was a hit when it was first published over 10 years ago. If you want to know more about Continuous Integration and Continuous Deployment this book is a must-read. It unfolds and explains multiple techniques and approaches to improve the delivery of software. You’ll learn not only what to do to improve how you develop software. You’ll also learn why some techniques work while others don’t. Such knowledge helps immensely with managing the backlog of non-functional features.

Rework

The guidance provided by Jason Fried and David Hansson will help you understand a lot on how a company can and should work. In my opinion the advice helps in surprising ways. Not only in the context of work! Since pandemic started many of us switched to working from home. The book shares some tips on how a company and more importantly people can work remotely efficiently.

Release It!: Design and Deploy Production-Ready Software

Have you ever heard a dreadful story about a software system bringing a major enterprise to its knees? Have you ever wondered what it takes for software to be mature and reliable? If so you’ll love this piece. With a mixture of technical insights and best practices coloured with production stories, this book reads like a novel. It explains in-depth multiple techniques that improve the resilience of software systems. Circuit breaker, bulkhead and redundancy will be your bread and butter after you get yourself familiar with what Michael T. Nygard describes.

What are the books that you think are worth reading? Please post comments with your recommendations!

Autoscaling AWS EKS cluster with custom Gitlab runner configuration

Wed, 17 Jun 2020 00:00:00 GMT

Gitlab offers a robust CI/CD solution suitable for most projects. The documentation depicts the easiest way to connect your Kubernetes cluster to Gitlab. Once connected Gitlab can install and configure Gitlab runner.

However, such approach does not allow for many adjustments to Gitlab runner configuration. For example when using Kubernetes executor providing build level resource requests and limits does not work.

In this post we’ll use aws-cdk to:

provision an AWS EKS cluster
install cluster autoscaler
install Gitlab runner using Helm chart

Prerequisite

Create an aws-cdk typescript project with necessary dependencies using

mkdir cicd && cd cicd
npx cdk init --language typescript
npm i --save @aws-cdk/aws-eks

Create EKS Cluster with aws-cdk

First we add an AWS EKS cluster with a managed node group. A managed node group provides a convenient way to provision and manage nodes.

export class CICDStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const clusterAdmin = new iam.Role(this, "Cluster Master Role", {
      assumedBy: new iam.AccountRootPrincipal(),
    });

    const clusterName = "cicd";
    const cluster = new eks.Cluster(this, "CICD Cluster", {
      mastersRole: clusterAdmin,
      clusterName: clusterName,
      defaultCapacity: 0,
    });

    const mainNodeGroup = cluster.addNodegroup("main", {
      desiredSize: 1,
      instanceType: ec2.InstanceType.of(InstanceClass.M5, InstanceSize.XLARGE2),
      nodegroupName: "main",
      maxSize: 10
    });

    ...

  }
}

The clusterAdmin IAM Role is going to be use to manage the cluster with kubectl. We do not need it to install and configure Gitlab runner. However, it is essential for debugging purposes.

Install Gitlab runner using Helm chart

For our EKS cluster to perform any CI work we need to install Gitlab runner. Thankfully Gitlab provides an official Helm chart.

cluster.addChart("gitlab-runner", {
  chart: "gitlab-runner",
  repository: "https://charts.gitlab.io/",
  version: "v0.17.1",
  wait: true,
  values: {
    gitlabUrl: "https://gitlab.com/",
    runnerRegistrationToken: "<runner registration token>",
    rbac: {
      create: true,
    },
    runners: {
      // https://gitlab.com/gitlab-org/charts/gitlab-runner/blob/master/values.yaml
      // required for dind
      privileged: true,
      builds: {
        cpuRequests: "1",
        cpuRequestsOverwriteMaxAllowed: "16",
        cpuLimitOverwriteMaxAllowed: "16",
        memoryRequests: "4Gi",
        memoryLimitOverwriteMaxAllowed: "16Gi",
        memoryRequestsOverwriteMaxAllowed: "16Gi",
      },
    },
  },
});

For the Gitlab runner to register in Gitlab server we need to get a runner registration token. You can find the runner registration token on a project or group CI/CD settings page. In the above example I passed the value as is to the runnerRegistrationToken setting. However, you should not commit that into source control and instead use an environment variable, or a parameter store reference.

Note that we have configured a couple of values defined in the Helm chart. Most notably we set the required cpu*OverwriteMaxAllowed and memory*OverwriteMaxAllowed options. We have to configure them otherwise it will not be possible to set Kubernetes CPU and memory allocations for requests and limits on the .gitlab-ci.yml. With the above set we can request a certain amount of CPU and memory for build:

 variables:
   KUBERNETES_CPU_REQUEST: 3
   KUBERNETES_CPU_LIMIT: 5
   KUBERNETES_MEMORY_REQUEST: 2Gi
   KUBERNETES_MEMORY_LIMIT: 4Gi

Install Cluster autoscaler using aws-cdk construct

In order for our AWS EKS cluster to scale dynamically we need to install Cluster autoscaler. With autoscaler installed, whenever the amount of declared requested CPU or memory resources exceeds those available on nodes, a new node will be added to the cluster.

I’ve wrapped the Cluster autoscaler setup into aws-cdk construct With the ClusterAutoscaler construct we can simply enhance our EKS Cluster by adding the following to the CICDStack definition:

new ClusterAutoscaler(this, "cluster-autoscaler", {
  cluster: cluster,
  nodeGroups: [mainNodeGroup],
  clusterName,
});

Connect to the cluster with kubectl

After a deployment the new Gitlab runner becomes available for a project or Gitlab group. The cdk deploy command will also output a hint on how to configure kubectl to connect to the cluster. It will look somewhat similar to the example below:

aws eks update-kubeconfig --name cicd --region eu-west-1 \
  --role-arn arn:aws:iam::XXXXXXXXXXX:role/CICD-ClusterMasterRoleXXXX-XXXX

You only need to execute the above command once. It will store the configuration for kubectl on your local disk. Afterwards you can inspect the EKS Kubernetes cluster with kubectl e.g:

> kubectl get pods 
NAME                                                              READY   STATUS    RESTARTS   AGE
cicdcicdclusterchartgitlabrunnerdXXXXX-gitlab-runner-XXXXXXXXXX   1/1     Running   0          26h

You can find full working example under the github repository.

Deploy aws-cdk app to different environments

Sun, 31 May 2020 00:00:00 GMT

Infrastructure as a code greatly improves traceability, repeatability and thus overall quality of the software systems built with it. With a correct setup development teams can now more easily spin up short-lived clones of systems or their parts. This gives a profound boost in both productivity but also ability to run automated regression or performance tests.

In this post I will show how to use aws-cdk to deploy multiple copies of the same application:

1. Using multiple AWS accounts.

2. Using a single AWS account.

Using multiple AWS accounts to deploy copies of the same aws-cdk application

By default, every aws-cdk Stack you create is environment-agnostic. You can deploy an environment-agnostic stack with any AWS account granted you have enough entitlements. Note that AWS account does not mean IAM user. A single AWS account can have multiple IAM users.

The cdk deploy command uses specified AWS profile or a default one.

Let us start with a bare bone app multiple-accounts generated with:

mkdir multiple-accounts && cd multiple-accounts
npx cdk init --language typescript

To make our example more entertaining I added a single S3 Bucket to the MultipleAccountsStack:

import * as cdk from "@aws-cdk/core";
import * as s3 from "@aws-cdk/aws-s3";

export class MultipleAccountsStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    new s3.Bucket(this, "MyBucket");
  }
}

I also created 2 AWS profiles for 2 AWS accounts, using aws configure, named blog-staging and blog-production for demonstration.

Deploy aws-cdk app using explicitly specified profile

For start let’s use --profile parameter to deploy our stacks:

> npx cdk deploy --profile blog-staging

MultipleAccountsStack: deploying...
MultipleAccountsStack: creating CloudFormation changeset...
 ...

 ✅  MultipleAccountsStack

Stack ARN:
arn:aws:cloudformation:eu-central-1:XXXXXXXXXX:stack/MultipleAccountsStack/0d3c7530-a318-11ea-ab8d-02b8022d5b40

> npx cdk deploy --profile blog-production

MultipleAccountsStack: deploying...
MultipleAccountsStack: creating CloudFormation changeset...
...

 ✅  MultipleAccountsStack

Stack ARN:
arn:aws:cloudformation:eu-central-1:YYYYYYYYYY:stack/MultipleAccountsStack/3a5cc690-a319-11ea-b00b-023cc2de05b4

Note how XXXXXXXXXX and YYYYYYYYYY identifies, which AWS account deployed the stack. The stack has the same name in both cases: MultipleAccountsStack

You can specify the AWS profile to use with AWS_PROFILE environment variable:

AWS_PROFILE=blog-production npx cdk deploy

Last but not least the standard AWS CLI environment variables work as well:

export AWS_ACCESS_KEY_ID=xxxx
export AWS_SECRET_ACCESS_KEY=yyy
export AWS_DEFAULT_REGION=eu-central-1
npx cdk deploy

Please refer to the documentation for further details.

Benefits of using multiple AWS accounts to deploy aws-cdk app

The first advantage is how using multiple AWS accounts keeps aws-cdk code simple. The infrastructure code ignores details of the environment it runs in. This makes it possible to re-use it in other contexts. This approach also improves security. With multiple AWS accounts we minimise risk of aws-cdk installations affecting each other.

Downsides of using multiple AWS accounts to deploy aws-cdk app

The following example presents a first downside, or in reality a gotcha easy to miss. Say someone not well versed in AWS services decided to name the bucket:

new s3.Bucket(this, "MyBucket", {
  bucketName: "important-assets",
});

It will work fine when deployed for the first time. However, as soon as we try to deploy it to a second AWS account we get:

> npx cdk deploy --profile blog-production
(node:61539) ExperimentalWarning: Conditional exports is an experimental feature. This feature could change at any time
MultipleAccountsStack: deploying...
MultipleAccountsStack: creating CloudFormation changeset...
 0/3 | 10:56:27 AM | CREATE_IN_PROGRESS   | AWS::S3::Bucket    | MyBucket (MyBucketF68F3FF0)
 0/3 | 10:56:27 AM | CREATE_IN_PROGRESS   | AWS::CDK::Metadata | CDKMetadata
 1/3 | 10:56:27 AM | CREATE_FAILED        | AWS::S3::Bucket    | MyBucket (MyBucketF68F3FF0) important-assets already exists

Some AWS resources names need a unique name in a given region across all AWS accounts.

The difficulty to create AWS accounts on a developer whim is the second downside. Remember that want to re-create environments with changed parts. Forcing a new AWS account may turn out problematic and create a management headache too big to ignore.

Using single AWS account to deploy copies of the same aws-cdk application

As we saw above by default the Stack name gets derived from the class name. Thankfully one can easily override that behaviour. In my projects I prefer using a DEPLOY_ENV environment variable to specify the deployment target.

DEPLOY_ENV=staging npx cdk deploy

We need to adjust our app definition so that DEPLOY_ENV is recognized.

Using DEPLOY_ENV variable inside aws-cdk entry point

The aws-cdk app typically resided as bin folder. In our example it is the bin/multiple-accounts.ts file. With the following adjustment we can deploy our app to many targets:

#!/usr/bin/env node
import "source-map-support/register";
import * as cdk from "@aws-cdk/core";
import { MultipleAccountsStack } from "../lib/multiple-accounts-stack";

const deployEnv = process.env.DEPLOY_ENV || "dev";
const app = new cdk.App();
new MultipleAccountsStack(app, deployEnv + "-MultipleAccountsStack");

During deployment aws-cdk will create a CloudFormation stack named ${process.env.DEPLOY_ENV}-MultipleAccountsStack:

Using DEPLOY_ENV consistently inside aws-cdk application

As I mentioned above one can easily fall into unique name traps. In order to avoid naming conflicts and to apply consistent naming scheme I often define a helper similar to the following.

function deployEnv() {
  return process.env.DEPLOY_ENV || "dev";
}

function envSpecific(logicalName: string | Function) {
  const suffix =
    typeof logicalName === "function" 
    ? logicalName.name 
    : logicalName;

  return `${deployEnv()}-${suffix}`;
}

export class MultipleAccountsStack extends cdk.Stack {
  constructor(scope: cdk.Construct, props?: cdk.StackProps) {
    super(scope, envSpecific(MultipleAccountsStack), props);

    new s3.Bucket(this, "MyBucket", {
      bucketName: envSpecific("important-assets"),
    });
  }
}

With the above change our cdk app entry point no longer decides how to name the stack. Note however, that you can still allow passing the name through constructor when you need it.

Using envSpecific helper keeps important resource names consistent. More importantly it allows deploying as many copies of aws-cdk app as we like.

Combine AWS Step Functions with CloudWatch Events using aws-cdk

Wed, 20 May 2020 00:00:00 GMT

AWS Step Functions allow one to execute & coordinate long-running processes. Step Functions fall into serverless AWS services, and the platform manages the function execution state completely.

In the example below we will use the following AWS services:

aws-cdk
Step Function
Lambda

The example demonstrates how Step Functions manage execution of a process, which involves external events e.g. human interaction.

Define step function

I will use blank infrastructure project created with

mkdir step-function && cd step-function
npx cdk init --language typescript

Next we need to add required packages:

cd step-function
npm i --save @aws-cdk/aws-stepfunctions \
  @aws-cdk/aws-stepfunctions-tasks \
  @aws-cdk/aws-lambda-nodejs \
  @aws-cdk/aws-dynamodb

We start with a state machine that contains a single Wait step as follows.

import { WaitTime } from "@aws-cdk/aws-stepfunctions";
import { Duration } from "@aws-cdk/core";
import * as cdk from "@aws-cdk/core";
import * as sfn from "@aws-cdk/aws-stepfunctions";

export class StepFunctionStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const wait = new sfn.Wait(this, "Wait for Event", {
      time: WaitTime.duration(Duration.seconds(5)),
    });

    new sfn.StateMachine(this, "Step Function", {
      definition: wait,
    });
  }
}

We deploy it with the usual aws-cdk deploy invocation

AWS_PROFILE=my-aws-profile npm run cdk deploy

After a few moments we can find our brand new AWS Step Function visual representation in AWS Web Console:

Define workflow in AWS Step Function

In order to make our example more interesting let’s add a bit complexity to it. Imagine we handle a document flow, which requires two separate actors to sign it off.

The actors act independently. The first signer, Alice, signs documents quickly. The other, Bob, can take a couple of days to process a document.

Once both actors sign the document our flow succeeds. However, if we do not collect the signatures until some time elapses the flow fails.

In order for the `@aws-cdk/aws-lambda-nodejs` to work properly you need to have Docker running

Alice’s part consists of a single task along with a lambda function.

const signDocument = new lambda.NodejsFunction(this, "Sign Document", {
  entry: __dirname + "/sign-document.ts",
  handler: "signDocument",
});

const aliceSignature = new sfn.Task(this, "Alice Signature", {
  task: new tasks.RunLambdaTask(signDocument, {
    payload: sfn.TaskInput.fromObject({
      signer: "Alice",
      documentId: Data.stringAt("$.documentId"),
    }),
  }),
});

The signDocument lambda function handler defined in step-function/lib/sign-document.ts looks as follows:

import { Handler } from "aws-lambda";

const delay = (millis: number) =>
  new Promise((resolve) => setTimeout(resolve, millis));

export const signDocument: Handler<{
  documentId: string;
  signer: string;
}> = async (event) => {
  await delay(1000);
  console.log("Sign document", event);
  return {
    result: `${event.signer} signed ${event.documentId}`,
  };
};

Please note that the lambda function can be async as long as it completes within the regular AWS Lambda function rules.

Bob’s signature is a more complex job. Since Bob can take several days to process a request to sign a document we cannot simply use lambda.

In order to solve that we will split the processing into 2 lambda functions using ServiceIntegrationPattern.WAIT_FOR_TASK_TOKEN. The first one will receive step function token and store it in a DynamoDb table. That simulates reaching out to Bob e.g. through email.

The second one will retrieve the token from the DynamoDb table and mark the Step function task as complete.

const signRequestsTable = new dynamodb.Table(this, "Signature Requests", {
  billingMode: BillingMode.PAY_PER_REQUEST,
  partitionKey: { name: "documentId", type: AttributeType.STRING },
});

const requestDocumentSignature = new lambda.NodejsFunction(
  this,
  "Request Document Signature",
  {
    entry: __dirname + "/sign-document.ts",
    handler: "requestDocumentSignature",
    environment: {
      signRequestsTableName: signRequestsTable.tableName,
    },
  }
);

const completeSignatureRequest = new lambda.NodejsFunction(
  this,
  "Complete Document Signature",
  {
    entry: __dirname + "/sign-document.ts",
    handler: "completeSignatureRequest",
    environment: {
      signRequestsTableName: signRequestsTable.tableName,
    },
  }
);

signRequestsTable.grantReadWriteData(requestDocumentSignature);
signRequestsTable.grantReadWriteData(completeSignatureRequest);

const bobSignature = new sfn.Task(this, "Bob Signature", {
  task: new tasks.RunLambdaTask(requestDocumentSignature, {
    integrationPattern: ServiceIntegrationPattern.WAIT_FOR_TASK_TOKEN,
    payload: sfn.TaskInput.fromObject({
      stepFunctionToken: sfn.Context.taskToken,
      signer: "Bob",
      documentId: Data.stringAt("$.documentId"),
    }),
  }),
});

Last but not least we need to allow the completeSignatureRequest to adjust step function state through IAM Policy Statement.

const waitForSignaturesTimeout = Duration.seconds(60);
const stateMachine = new sfn.StateMachine(this, "Step Function", {
  definition: flow,
  timeout: waitForSignaturesTimeout,
});

completeSignatureRequest.addToRolePolicy(
  new iam.PolicyStatement({
    actions: ["states:SendTaskSuccess", "states:SendTaskFailure"],
    resources: [stateMachine.stateMachineArn],
  })
);

After we deploy our updated stack the function definition should look as follows:

Call Step Function callback from AWS Lambda

For our workflow to complete we need to implement Bob’s lambda part. Let’s start with a function that simulates starting a long-running business process. The important bit in sendSignatureRequest is that it stores state function task token. We will use the token later on to call AWS Step Function back. The callback will include result. In our case a fake signature of a document.

First add 2 more packages to our project:

npm i --save @types/aws-lambda aws-sdk

Then update the AWS Lambda sendSignatureRequest definition:

import { Handler } from "aws-lambda";
const DynamoDb = require("aws-sdk/clients/dynamodb");

interface SignDocumentEvent {
  documentId: string;
  signer: string;
  stepFunctionToken: string;
}

const documentClient = new DynamoDb.DocumentClient();
const signRequestsTableName = process.env.signRequestsTableName;

export const requestDocumentSignature: Handler<SignDocumentEvent> = async (
  event
) => {
  console.log("Request signature for", event);

  await documentClient
    .put({
      TableName: signRequestsTableName,
      Item: event,
    })
    .promise();
};

The last missing piece is to implement the completeSignatureRequest lambda function. The important bit is to call sendTaskSuccess to indicate that processing completed successfully.

import { Handler } from "aws-lambda";
const StepFunctions = require("aws-sdk/clients/stepfunctions");
const DynamoDb = require("aws-sdk/clients/dynamodb");
const documentClient = new DynamoDb.DocumentClient();
const signRequestsTableName = process.env.signRequestsTableName;
const stepFunctions = new StepFunctions();
export const completeSignatureRequest: Handler<{ documentId: string }> = async (
  event
) => {
  const tableItem = await documentClient
    .get({
      TableName: signRequestsTableName,
      Key: { documentId: event.documentId },
    })
    .promise();

  const signRequest: SignDocumentEvent = tableItem.Item;
  console.log("Received signature for", signRequest);

  const output = JSON.stringify({
    signature: `${signRequest.signer} signed ${signRequest.documentId}`,
  });

  await stepFunctions
    .sendTaskSuccess({
      output: output,
      taskToken: signRequest.stepFunctionToken,
    })
    .promise();
};

Let’s test the execution by requesting a document with id equal to 3. As expected Alice signs the document after couple of seconds. However, for in order to get Bob’s signature we invoke complete signature lambda. This simulates an action that involves human interaction. Note that the whole process could take months to complete!

The AWS Step Function console depicts the current phase of processing pretty neatly.

You can find full working example in Github repository.

Leave a comment if you like the article and would like to see more!

HTTPs on CloudFront using Certificate Manager and aws-cdk

Tue, 05 May 2020 00:00:00 GMT

The static website deployed with aws-cdk to CloudFront post shows how to deploy static content with aws-cdk. We still miss a custom domain configuration for our website though. The following example shows how to:

setup a custom domain name for a CloudFront distribution
enable https using AWS issued trusted certificate

Get a trusted certificate from AWS Certificate Manager using aws-cdk

Using the example from previous post as a base, inside infrastructure directory, let’s add additional package:

npm i --save @aws-cdk/aws-certificatemanager

CloudFront can only use AWS Certificate Manager issued certificates inside us-east-1 region (N. Virginia). We can create our FrontendStack in us-east-1 region. Alternatively we can create another stack with the certificate only. Below I show how to use the second option.

...
import { ValidationMethod } from "@aws-cdk/aws-certificatemanager";
import * as cm from "@aws-cdk/aws-certificatemanager";

const domainName = "fast-static-website.miensol.pl";

export class CertificateStack extends cdk.Stack {
  constructor(scope: cdk.Construct) {
    super(scope, "CertificateStack", {
      env: { region: "us-east-1" },
    });

    const certificate = new cm.Certificate(this, "CustomDomainCertificate", {
      domainName: domainName,
      validationMethod: ValidationMethod.DNS,
    });

    const certificateArn = certificate.certificateArn;
    new CfnOutput(this, "CertificateArn", {
      value: certificateArn,
    });
  }
}
...

We request a certificate for a given domain name in us-east-1 region. Before AWS Certificate Manager issues a trusted TLS certificate it needs to verify that you own the domain. The validation happens using either:

email: AWS CM sends a message to admin email addresses for the domain
DNS: AWS CM requests a particular DNS record configuration

In the example above I use fast-static-website.miensol.pl with DNS validation method. Since I own the domain I can prove that to AWS Certificate Manager.

We also update the bin/infrastructure.ts file to include our new stack:

#!/usr/bin/env node
import "source-map-support/register";
import * as cdk from "@aws-cdk/core";
import { CertificateStack, FrontendStack } from "../lib/frontend-stack";

const app = new cdk.App();
new CertificateStack(app);
new FrontendStack(app);

Let’s trigger next deployment with:

npm run cdk deploy '*'

The process will wait for a certificate request validation. The requested DNS record will show up in the deployment command output:

> npm run cdk deploy
...
1/5 | 6:03:37 PM | CREATE_IN_PROGRESS   | AWS::CertificateManager::Certificate
 | CustomDomainCertificate (CustomDomainCertificateXXXXX) Content of DNS Record is: {Name: XXXXXX.fast-static-website.miensol.pl.,Type: CNAME,Value: XXXXX.auiqqraehs.acm-validations.aws.}
1/5 Currently in progress: CustomDomainCertificateXXXXX

You can also find the request in Certificate Manager inside AWS Web console. After you configure the DNS record, the process will continue. In the command output you should see a message similar to:

CertificateStack.CertificateArn = arn:aws:acm:us-east-1:XXXXXXX:certificate/XXXXX-XXXXX-XXXX-XXXX-XXXXXXXX

Please note the above value, we will need it in the following step.

Attach a certificate to CloudFront distribution using aws-cdk

We need to tell the CloudFront distribution to use the new certificate. Let’s modify the FrontendStack to accept parameters:


interface FrontendProps {
  certificateArn: string;
}

export class FrontendStack extends cdk.Stack {
  constructor(scope: cdk.Construct, props: FrontendProps) {
    super(scope, "FrontendStack");

    const frontBucket = new s3.Bucket(this, "FrontendBucket", {
      websiteIndexDocument: "index.html",
      publicReadAccess: true,
    });

    const distribution = new cloudfront.CloudFrontWebDistribution(
      this,
      "Distribution",
      {
        viewerCertificate: {
          aliases: [domainName],
          props: {
            acmCertificateArn: props.certificateArn,
            sslSupportMethod: "sni-only",
          },
        },
...

In the viewerCertificate section we specify domain name as well as certificate CloudFront distribution should use. Next we pass CertificateStack.CertificateArn value noted above as a parameter:

const app = new cdk.App();
new CertificateStack(app);
new FrontendStack(app, {
  certificateArn:
    "arn:aws:acm:us-east-1:XXXXXXX:certificate/XXXXX-XXXXX-XXXX-XXXX-XXXXXXXX",
});

The deployment will take a while to complete since the CF distribution has to update. We can use that time to configure fast-static-website.miensol.pl DNS record. The record needs to point to FrontendStack.DistributionDomainName. I use CloudFlare so in my case this means creating a CNAME record:

We can now load our static website using HTTPS!

It is a shame that it is not possible to create cross region references using aws-cdk:

There is not, unfortunately. You’re going to have to thread the API URL across regions manually. That means:

Deploy the ApiStackIreland first. Note the API URL (maybe make it an Output so that it’s easy to get to). Deploy the EdgeStack and plug in the value of API URL from the first step. There are a number of ways you could do this, and honestly I’m not sure which one to recommend.

I guess a pragmatic one would be to use this.node.getContext() and pass the value on the command-line when deploying the second stack (using the -c KEY=VALUE fflag), or put it into cdk.json under the context: {} section.

The full working example is available in the github repository.

Deploy fast, static website to AWS like a pro

Sun, 03 May 2020 00:00:00 GMT

In this post we’ll see how to create and deploy a static website using IaC using the following technologies:

aws-cdk
AWS CloudFormation
AWS Lambda
AWS S3

Have a static website

For the sake of simplicity we’ll use Create React App in a directory called my-project:

cd my-project
npx create-react-app frontend --template typescript

Create the static website content inside frontend/build directory with:

cd frontend && yarn build

Add aws-cdk to the project

Create infrastructure project with aws-cdk:

mkdir infrastructure && cd infrastructure
npx cdk init --language typescript

Note that in order to deploy you need to have AWS API credentials configured. I prefer using AWS profiles, which you can set up using AWS CLI:

aws configure --profile my-profile-name

The command will ask you couple of questions. Please find the AWS Access Key Id and AWS Secret Access Key for your AWS account in AWS Web Console IAM section. The AWS CLI stores configuration under ~/.aws (Linux or Mac) or %USERPROFILE%\.aws (Windows).

Please follow aws-cdk Getting Started to get more details.

Deploy static content deployment to S3 website

We will deploy static content to AWS using @aws-cdk/aws-s3-deployment package. We need to add it to our project:

cd infrastructure
npm i --save @aws-cdk/aws-s3-deployment

Next, let’s tell aws-cdk to deploy contents of frontend/build directory to AWS. In our template we have InfrastructureStack available under lib/infrastructure-stack.ts. Since it makes sense to have more focused CloudFormation stacks let’s rename it to FrontendStackStack.

import * as cdk from '@aws-cdk/core';
import * as s3 from '@aws-cdk/aws-s3'
import * as s3deploy from '@aws-cdk/aws-s3-deployment'
import * as cloudfront from '@aws-cdk/aws-cloudfront'
import * as path from 'path'

export class FrontendStack extends cdk.Stack {
  constructor(scope: cdk.Construct) {
    super(scope, 'FrontendStack');
  }
}

All files, which our static content consists of, will be stored on S3, so we need to create a bucket there:

super(scope, 'FrontendStack');

const frontBucket = new s3.Bucket(this, 'FrontendBucket', {
  websiteIndexDocument: 'index.html',
  publicReadAccess: true
});

Next, we need to tell @aws-cdk/aws-s3-deployment to push contents of frontent/build to the bucket created above:

new s3deploy.BucketDeployment(this, 'DeployWebsite', {
  sources: [s3deploy.Source.asset(
    path.join(__dirname, '..', '..', 'frontend', 'build')
  )],
  destinationBucket: frontBucket
});

Since our aws-cdk stack uses assets we need to bootstrap our AWS account before first deployment:

cd infrastructure
AWS_PROFILE=my-profile-name npx cdk bootstrap

Once the above command completes we can now finally deploy our website:

cd infrastructure
AWS_PROFILE=my-profile-name npm run cdk deploy

The command will prompt for confirmation before it creates the necessary security rules. You will see what resources get created or modified while the command runs. Afterwards all resources created are part of AWS CloudFormation FrontendStack stack. Most importantly the command copied all files to S3 Bucket, which in turn can host the website.

One you find the S3 bucket created in the AWS CloudFormation stack:

Your website is now available under a URL similar to:

http://frontendstack-frontendbucketefe2e19c-ixsoxq891rl1.s3-website.eu-central-1.amazonaws.com/

Improve performance of static website with CloudFront

The website loads files directly from S3 bucket. The S3 bucket, by default, lives inside a single AWS Region, which means that visitors from far away may suffer from subpar loading speeds.

For production setup we can use AWS CloudFront CDN, which distributes our content across over 100 locations around the globe. Using this solution with @aws-cdk/aws-s3-deployment is a piece of cake:

const distribution = new cloudfront.CloudFrontWebDistribution(this, 'Distribution', {
  originConfigs: [{
    s3OriginSource: {
      s3BucketSource: frontBucket
    },
    behaviors : [ {isDefaultBehavior: true}]
  }]
});

new cdk.CfnOutput(this, 'DistributionDomainName', {
  value: distribution.domainName
})

new s3deploy.BucketDeployment(this, 'DeployWebsite', {
  sources: [s3deploy.Source.asset(
      path.join(__dirname, '..', '..', 'frontend', 'build')
  )],
  destinationBucket: frontBucket,
  distribution: distribution,
});

After you update the stack with AWS_PROFILE=my-profile-name npm run cdk deploy you will find a new CloudFront distribution created. Note that it make take few minutes to complete. We added CfnOutput to make it easier to find the CloudFront distribution autogenerated domain name.

Outputs:
FrontendStack.DistributionDomainName = d3a9wcc4fxw2d.cloudfront.net

Now the static website uses global content delivery network!

You can find full example used in this post in the Github repository

How to convert an express app to AWS Lambda?

Tue, 29 May 2018 00:00:00 GMT

In this post we will see how to convert an existing express application to AWS Lambda. This can help reduce AWS bill even by an order of magnitude. We will also use cloudform to describe the CloudFormation stack.

An express app

For our example to be complete we need an express application. Let’s use a single index.js file to define it:

const express = require('express');

function apiRoutes(){
    const routes = new express.Router();

    routes.get('/v1/version', (req, res) => res.send({version: '1'}));

    routes.post('/v1/echo', (req, res) => res.send({...req.body}));

    return routes;
}


const app = express()
    .use(express.json())
    .use(apiRoutes());

app.listen(3000, () => console.log(`Listening on 3000`));

The above application has only 2 endpoints:

GET /v1/version returns API version
POST /v1/echo sends back the request body

We start the application as any other node app with node index.js.

Convert the express app to AWS Lambda

The AWS Lambda Node.js runtime model differs from a simple node fileName.js invocation. For the AWS Lambda to invoke our application code we need to structure it appropriately. Thankfully the aws-serverless-express module adapts express to AWS Lambda Node.js runtime model.

Let’s adapt our index.js file to support aws-serverless-express:

const isInLambda = !!process.env.LAMBDA_TASK_ROOT;
if (isInLambda) {
    const serverlessExpress = require('aws-serverless-express');
    const server = serverlessExpress.createServer(app);
    exports.main = (event, context) => serverlessExpress.proxy(server, event, context)
} else {
    app.listen(3000, () => console.log(`Listening on 3000`));
}

The main function is called by the AWS Lambda Node.js runtime. Note that we’ve used LAMBDA_TASK_ROOT environment variable to detect if the app is running inside AWS Lambda. It is better to split the express application setup and listen call into separate files and use 2 different main files e.g. development.js calling listen(port) and lambda.js using aws-serverless-express. However, this would complicate our example unnecessarily.

Deploy the express app to AWS Lambda

I already showed how we can deploy lambda with cloudform. We will use the previous example as a base:

import cloudform, { Lambda, IAM, Fn, ApiGateway, Refs,  } from 'cloudform';
import { FunctionProperties } from 'cloudform/types/lambda/function';
import { readFileSync } from 'fs';

const
    LambdaExecutionRole = 'LambdaExecutionRole',
    ExpressMain = 'ExpressMain',
    RestApi = 'RestApi',
    RestApiMainResource = 'RestApiMainResource',
    PackageKey = 'PackageKey',
    RestApiDeployment = 'RestApiDeployment',
    RestApiMethod = 'RestApiMethod';

export default cloudform({
    Parameters: {
        PackageKey: {
            Type: 'String',
            Default: 'express-lambda.zip'
        }
    },
    Resources: {
        [ExpressMain]: new Lambda.Function({
            Code: { S3Bucket: 'bright-tmp', S3Key: Fn.Ref(PackageKey) },
            Handler: "index.main",
            Role: Fn.GetAtt(LambdaExecutionRole, "Arn"),
            Runtime: "nodejs6.10"
        }),
        [LambdaExecutionRole]: new IAM.Role({
            AssumeRolePolicyDocument: {
                Statement: [{
                    Effect: "Allow",
                    Principal: { Service: ["lambda.amazonaws.com"] },
                    Action: ["sts:AssumeRole"]
                }]
            },
            ManagedPolicyArns: ["arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"]
        }),
        PermissionToInvokeLambda: new Lambda.Permission({
            Action: 'lambda:InvokeFunction',
            FunctionName: Fn.GetAtt(ExpressMain, 'Arn'),
            Principal: 'apigateway.amazonaws.com',
            SourceArn: Fn.Join('', [
                'arn:aws:execute-api:', Refs.Region, ':', Refs.AccountId, ':', Fn.Ref(RestApi), '/*']
            )
        })
        ... // Rest Api Gateway see below
    }
})

For the AWS Lambda function source we use a zip file hosted in a S3 Bucket named packages at key specified in the template parameter PackageKey. The package zip file must contain all application source code including node_modules. We also define PermissionToInvokeLambda so that API Gateway can invoke the lambda function.

Add API Gateway to call AWS Lambda

To be able to invoke the AWS Lambda function using HTTP protocol we will use API Gateway. There are multiple ways of setting up the API gateway but we will use an appoach that is simplest in my opinion.

[RestApi]: new ApiGateway.RestApi({ Name: "Express API" }),
[RestApiMainResource]: new ApiGateway.Resource({
    RestApiId: Fn.Ref(RestApi),
    ParentId: Fn.GetAtt(RestApi, 'RootResourceId'),
    PathPart: "{proxy+}",
}),
[RestApiMethod]: new ApiGateway.Method({
    HttpMethod: 'ANY',
    ResourceId: Fn.Ref(RestApiMainResource),
    RestApiId: Fn.Ref(RestApi),
    AuthorizationType: 'NONE',
    Integration: {
        Type: "AWS_PROXY",
        IntegrationHttpMethod: "POST",
        Uri: Fn.Sub("arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ExpressMain.Arn}/invocations", {})
    }
}),
[RestApiDeployment]: new ApiGateway.Deployment({
    RestApiId: Fn.Ref(RestApi),
    StageName: 'test'
})

We first define a RestApi which is a collection of resources with various configuration options describing the API. Next is the RestApiMainResource which depicts a single REST resource. However, in our case we use wildcard PathPart to match all paths. This way we can have a single resource encompassing all API endpoints. For the RestApiMainResource we also need to define a single RestApiMethod. Note that the HttpMethod is set to ANY which matches all verbs. The Integration specifies that we would like to proxy requests to the ExpressMain AWS Lambda function. Last but not least we define a RestApiDeployment so that we have an URL to call the API.

It is handy to define Outputs in our template so that we can easily access the API url:

Outputs: {
    RestApiUrl: {
        Value: Fn.Join('', [Fn.Ref(RestApi), '.execute-api.', Refs.Region, '.amazonaws.com/test'])
    }
}

Test API Gateway calling AWS Lambda

With our cloudform template deployed with a single command:

aws cloudformation update-stack \
  --stack-name lambda-example \
  --capabilities CAPABILITY_IAM \
  --template-body file://<(./node_modules/.bin/cloudform aws-template.ts)

We can also fetch the API url:

API_URL=$(aws cloudformation describe-stacks \
  --stack-name lambda-example \
  --query 'Stacks[0].Outputs[?OutputKey==`RestApiUrl`].OutputValue' \
  --output text)

Finally, with the help of httpie we can test our API:

> http https://${API_URL}/v1/version

HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 15
Content-Type: application/json; charset=utf-8
Date: Mon, 28 May 2018 19:58:13 GMT
Via: 1.1 XXXXXXXXXXXXX.cloudfront.net (CloudFront)
X-Amz-Cf-Id: XXXXXXXXXXXXX==
X-Amzn-Trace-Id: Root=1-5b0c5f54-806e190d437c7d31a4a0d4ba
X-Cache: Miss from cloudfront
etag: W/"f-sHigu4BMVa0IJ0LR3NDJ5y8l4sc"
x-amz-apigw-id: HnPVQH4yjoEF8-w=
x-amzn-Remapped-connection: close
x-amzn-Remapped-content-length: 15
x-amzn-Remapped-date: Mon, 28 May 2018 19:58:13 GMT
x-amzn-RequestId: 73eff8a6-62b1-11e8-907c-79117668835e
x-powered-by: Express

{
    "version": "1"
}

And a POST to the echo endpoint:

http https://${API_URL}/v1/echo message="Hello 👋 My name is Piotr"

HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 41
Content-Type: application/json; charset=utf-8
Date: Mon, 28 May 2018 20:00:31 GMT
Via: 1.1 XXXXXXXXXXXXX.cloudfront.net (CloudFront)
X-Amz-Cf-Id: XXXXXXXXXXXXX==
X-Amzn-Trace-Id: Root=1-5b0c5fdf-802178cceb5160684ef2cc34
X-Cache: Miss from cloudfront
etag: W/"29-SKqhJThIfjmVId6IIeTilD7Mkk0"
x-amz-apigw-id: HnPq5HeJjoEFuRQ=
x-amzn-Remapped-connection: close
x-amzn-Remapped-content-length: 41
x-amzn-Remapped-date: Mon, 28 May 2018 20:00:31 GMT
x-amzn-RequestId: c67b41c0-62b1-11e8-a23f-c7cbebde15f2
x-powered-by: Express

{
    "message": "Hello 👋 My name is Piotr"
}

Reduced infrastructure cost

With the above setup we no longer have to pay for an always running EC2 instance. Our monthly bill depends on the number of requests that are executed against the exposed API. More than that, we get much better scalability characteristics, especially when we have nonlinear request rates.

TypeScript tips from Kotlin developer - const

Wed, 11 Apr 2018 00:00:00 GMT

Nowadays I mostly code in Kotlin programming language. I got interested in it when I started working on a Java 7 codebase and quickly got fed up with the language. At Bright Inventions we often use TypeScript for both back-end, front-end and mobile development so I thought I would share some thoughts and tips for TypeScript learned while using Kotlin. In particular this post is about constant variables.

Use `const` whenever possible

Using immutable variables aids reasoning about the flow and state of a program. It helps compiler to provide more intelligent hints especially while dealing with nullable types.

In Kotlin a val keyword denotes a variable which value does not change after initialization e.g.:

val x: Int
val y = 3
x = 2
x = 20 // Error: Val cannot be reassigned
y = 30 // Error: Val cannot be reassigned

In TypeScript such a situation is handled with const:

const x: number // Error: 'const' declarations must be initialized
const y: number = 3

y = 30 // Error: Cannot assign to 'y' because it is constant or read-only property

Compilers love `const`

The first way in which the compiler gets smarter when we use const are null checks. When you enable strictNullChecks, which you should, both Kotlin and TypeScript compiler understand if something can or cannot be null.

const firstName: string | null = getFirstName()
let lastName: string | null = getLastName()

if (firstName !== null && lastName !== null) {
    setTimeout(() => {
        console.log(firstName.length)
        console.log(lastName.length) // Error: Object is possibly 'null'
    })
}

In the first 2 lines we declare firstName and lastName as holding string or null. The variables are initialized with the helper functions getFirstName and getLastName. After we check that firstName and lastName are definitely not null we fire some async function. We can safely use firstName.length. However, when we use lastName.length the compiler complains with Object is possibly 'null' error. This is because it is possible that in between the null check and the console.log statement something could change the lastName value. We might know that this is not true just by looking at the code. The compiler however, cannot be 100% sure in all cases. Thankfully we have const and we can share our knowledge with the compiler.

Compilers catch bugs with `const`

Because const and val can only be assigned once, compilers can prevent another class of bugs. Take a look at the code example blow. There is a bug 🐛 that could easily be avoided had we used const instead of let.

let firstName: string = person.firstName
let lastName: string = person.lastName

const parsed = parseFormData((data: {name: string }) => {
    let first: string | null, last: string | null
    let parts = data.name.split(' ')
    lastName = parts[0]
    firstName = parts[1]
    return { firstName: first, lastName: last }
})

if (parsed.firstName !== firstName || parsed.lastName !== lastName) { 
    // submit changes
}

You may have spotted the bug. Chances are though, especially if you are like me, that after long hours when the coffee level in your blood drops substantially below the required level, it will take long minutes to figure out the cause. There is a very easy remedy though. With firstName and lastName declared as constant variables the compiler catches bugs for us:

lastName = parts[0] // Error: Cannot assign 'lastName' because it is a constant or a read-only property
firstName = parts[1] // Error: Cannot assign 'firstName' because it is a constant or a read-only property

How to deploy Lambda function with CloudFormation?

Sun, 11 Mar 2018 00:00:00 GMT

Serverless deployments are popular these days. With a minimal cost you can have your own code wait and respond to various events. AWS Lambda, Azure Functions are just 2 examples of serverless offering from the biggest cloud providers. For a long time I had thought about them only in the context of ad-hoc setups not suitable for a long term development. This was until I found out that you can, with a little effort, version and deploy the serverless API just as a traditional back-end. In this post I am going to show how to deploy AWS Lambda functions with the help of the tool Adam created at Bright Inventions called cloudform.

Step 1: Define a template

Let’s install the cloudform npm i --save-dev cloudform and define a minimal template:

import cloudform, { Lambda, IAM, Fn } from 'cloudform';

const LambdaExecutionRole = 'LambdaExecutionRole';

cloudform({
    Resources: {
        HelloWorld: new Lambda.Function({
            Code: {
                ZipFile: "exports.wrtiteToConsole = function (event, context, callback){ console.log('Hello'); callback(null); }"
            },
            Handler: "index.wrtiteToConsole",
            Role: Fn.GetAtt(LambdaExecutionRole, "Arn"),
            Runtime: "nodejs6.10"
        }),
        [LambdaExecutionRole]: new IAM.Role({
            AssumeRolePolicyDocument: {
                Statement: [{
                    Effect: "Allow",
                    Principal: { Service: ["lambda.amazonaws.com"] },
                    Action: ["sts:AssumeRole"]
                }]
            },
            Path: "/",
            ManagedPolicyArns: ["arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"]
        })
    }
});

There are only 2 resources defined in the template. HelloWorld is the AWS Lambda function definition. We have set the Runtime property to use nodejs6.10 hence the Code.ZipFile contains a JavaScript code. Despite the name ZipFile should contain application source code in plain text. Obviously the ZipFile may only be used for the most simple functions. The S3Bucket and S3Key properties can be used to deploy a more involved function implementation. The Handler defines which function from Code the Lambda runtime should invoke. In case of ZipFile the first part is always index.

The Role is a required setting for every AWS Lambda function. It defines what entitlements the code invoked by the Lambda runtime has. In the above template we define a policy which can be assumed by lambda.amazonaws.com and references a pre-defined AWS managed policy AWSLambdaBasicExecutionRole. The AWSLambdaBasicExecutionRole makes it possible for the Lambda function logs to be pushed to Cloud Watch.

Step 2: Create the AWS Lambda function

Deploying our Lambda function using CloudFormation requires a single command:

aws cloudformation create-stack \
    --capabilities CAPABILITY_IAM \
    --stack-name lambda-example \
    --template-body file://<(node_modules/.bin/cloudform aws-template.ts)

The --capabilities CAPABILITY_IAM is required whenever the CloudFormation has to define Roles, Policies or related resources. The --template-body file://<(node_modules/.bin/cloudform aws-template.ts) instructs CloudFormation to use a template defined in a file. The <(...) is bash and zsh way to pass an output of a command to other program as if the output was a file. After waiting a bit for the invocation to complete we will see the following in the AWS Console:

It is possible to use the AWS Console editor to test and change the function. However, if we are to treat the serverless approach seriously we should not forget about the standard practices like versioning of our source code.

Step 3: Update and version AWS Lambda function

Since we have defined the AWS Lambda function using a cloudform template we can version it as any other code. The whole serverless infrastructure we use and configure is treated as a source code allowing for an easy replication of deployment environments, audit trail and change management. Let’s see how we can use cloudform to add another function that will be called by the first one.

import cloudform, { Lambda, IAM, Fn } from 'cloudform';
import { FunctionProperties } from 'cloudform/types/lambda/function';
import { readFileSync } from 'fs';

const
    LambdaExecutionRole = 'LambdaExecutionRole',
    Alice = 'Alice',
    Bob = 'Bob';

function lambdaFunction(
    functionCode: string,
    options?: Partial<FunctionProperties>) {
    return new Lambda.Function({
        Code: { ZipFile: functionCode },
        Handler: "index.main",
        Role: Fn.GetAtt(LambdaExecutionRole, "Arn"),
        Runtime: "nodejs6.10",
        ...options
    });
}

export default cloudform({
    Resources: {
        [Alice]: lambdaFunction(readFileSync('Alice.js', 'utf-8'), {
            Environment: {
                Variables: {
                    BobFunction: Fn.GetAtt(Bob, "Arn")
                }
            }
        }),
        [Bob]: lambdaFunction(readFileSync('Bob.js', 'utf-8')),
        ... // LambdaExecutionRole see below
    }
})

As you can see above, we’ve extracted a function lambdaFunction to simplify Lambda function declaration. Both Alice and Bob function’s bodies are defined in separate files. Interestingly Alice function, during the invocation, will have access to BobFunction environment variable pointing to Bob function ARN.

Our LambdaExecutionRole lacks a permission to invoke another Lambda function. Let’s fix that:

    [LambdaExecutionRole]: new IAM.Role({
        AssumeRolePolicyDocument: {
            Statement: [{
                Effect: "Allow",
                Principal: { Service: ["lambda.amazonaws.com"] },
                Action: ["sts:AssumeRole"]
            }]
        },
        Path: "/",
        ManagedPolicyArns: ["arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"],
        Policies: [{
            PolicyName: "AllowCallingOtherLambda",
            PolicyDocument: {
                Version: "2012-10-17",
                Statement: [{
                    Sid: "InvokeLambdaPermission",
                    Effect: "Allow",
                    Action: ["lambda:InvokeFunction"],
                    Resource: "*"
                }]
            }
        }]
    })

The Alice and Bob sources export main functions invoked by AWS Lambda runtime:

// Alice.ts
import aws from 'aws-sdk';

export function main(event: any, context: any, callback: Function) {
    new aws.Lambda().invoke({
        FunctionName: process.env.BobFunction!,
        Payload: JSON.stringify({message: "Hi!. I'm Alice."})
    }, (error: Error, data: any) => {
        const response = JSON.parse(data.Payload);
        console.log('FromBob', error || response)
        callback(error);
    });
}

// Bob.ts
export function main(event: any, context: any, callback: any) {
    console.log('FromAlice', event)
    callback(null, {message: "Hi! I'm Bob. Nice to meet you!"});
}

Since we now use TypeScript for Alice and Bob source we need to install the compiler npm i --save-dev typescript @types/node aws-sdk. Unfortunately currently in cloudform it is not possible to use custom tsconfig.json for the compilation. Fortunately we can invoke the compiler manually and use its outputs as any other Node.js source code. With a tsconfig.json:

{
  "compilerOptions": {
    "module": "commonjs",
    "target": "es6",
    "strict": true, 
    "strictPropertyInitialization": false,
    "esModuleInterop": true
  },
  "exclude": ["node_modules"],
  "compileOnSave": true
}

we can use the following commands to compile and deploy Lambda functions:

./node_modules/.bin/tsc

aws cloudformation update-stack \
    --capabilities CAPABILITY_IAM \
    --stack-name lambda-example \
    --template-body file://<(node -e "console.log((require('./aws-template').default))")

After the stack update completes we now should have 2 Lambda functions available. When we invoke the Alice function, we’ll see that the 2 AWS Lambda functions communicate:

START RequestId: 6a87c764-251e-11e8-b921-f9ca7649c7d7 Version: $LATEST
2018-03-11T11:22:00.615Z	6a87c764-251e-11e8-b921-f9ca7649c7d7	Alice says: Hi!. I'm Alice.
2018-03-11T11:22:01.676Z	6a87c764-251e-11e8-b921-f9ca7649c7d7	Bob says: Hi! I'm Bob. Nice to meet you!
END RequestId: 6a87c764-251e-11e8-b921-f9ca7649c7d7
REPORT RequestId: 6a87c764-251e-11e8-b921-f9ca7649c7d7	Duration: 1110.68 ms	Billed Duration: 1200 ms 	Memory Size: 128 MB	Max Memory Used: 34 MB

Serverless infrastructure offers endless possibilities. With cloudform we can take AWS Lambda development, change management and deployment to the next level.

How to call a load balanced ECS service?

Tue, 06 Mar 2018 00:00:00 GMT

A service running ECS can call plethora of AWS APIs. It can read messages from queues, publish messages to SNS topics, query a database. These are all valid ways to communicate with the service. However, often the most appropriate way is to call the service by an HTTP API. In this post I’ll describe how to configure an ECS service running inside VPC so that other services can call its API.

Deploy an ECS service to multiple hosts

Whenever we care about availability of a service running inside AWS, we need to have it running in at least 2 availability zones. For that reason when defining the ECS Cluster Auto Scaling Group we need to specify at least 2 VPC Subnets running in different availability zones.

"ECSMainCluster": {
    "Type": "AWS::ECS::Cluster",
    "Properties": { "ClusterName": "Main Cluster" }
},
"ECSAutoScalingGroup": {
    "Type": "AWS::AutoScaling::AutoScalingGroup",
    "Properties": {
        "VPCZoneIdentifier": [
            { "Ref": "PrivateASubnet" },
            { "Ref": "PrivateBSubnet" }
        ],
        "LaunchConfigurationName": { "Ref": "ContainerHostInstances" },
        "MinSize": "2",
        "MaxSize": "6",
        "DesiredCapacity": "2"                
    }
},
"ServiceA": {
    "Type": "AWS::ECS::Service",
    "Properties": {
        "Cluster": { "Ref": "ECSMainCluster" },
        "DesiredCount": 2,
        "LoadBalancers": [{
            "ContainerName": "serviceA",
            "ContainerPort": 8080,
            "TargetGroupArn": { "Ref": "ServiceAAlbTargetGroup" }
        }],
        "DeploymentConfiguration": { "MinimumHealthyPercent": 50 },
        "Role": { "Ref": "ECSServiceRole" },
        "TaskDefinition": { "Ref": "ServiceATask" }
    }
}

Notice how we are using 2 private subnets as VPCZoneIdentifier. The MinSize is also set to 2 which will cause both availability zones to have at least 1 instance running. For brevity subnets and VPC definitions are not included. You can find more details about how to configure the EC2 instances inside ECS cluster in my previous post.

The ServiceA deployed in ECSMainCluster is also specifying that at it has DesiredCount of 2 which instructs ECS to have at least 2 instances of the service running. The LoadBalancers and Role attributes are required for the load balanced setup. The ECSServiceRole must allow the ECS agent to make calls to the load balancer API. The LoadBalancers reference an ALB target group to which the running ECS task should be added.

A private load balancer

In order for us to be able to call an API exposed by ECS service running on multiple instances we will use an internal Application Load Balancer. By internal, I mean that the load balancer will not be accessible outside of the VPC.

"PrivateApiLoadBalancer": {
    "Type": "AWS::ElasticLoadBalancingV2::LoadBalancer",
    "Properties": {
        "Subnets": [
            { "Ref": "PrivateASubnet" },
            { "Ref": "PrivateBSubnet" }
        ],
        "Name": "PrivateApiLoadBalancer",
        "Scheme": "internal",
        "SecurityGroups": [{
            "Ref": "HttpHttpsProxySecurityGroup"
        }]
    }
},
"HttpHttpsProxySecurityGroup": {
    "Type": "AWS::EC2::SecurityGroup",
    "Properties": {
        "GroupDescription": "Enable http and https access",
        "VpcId": { "Ref": "VPC" },
        "SecurityGroupIngress": [{
            "IpProtocol": "tcp", "FromPort": "80", "ToPort": "80",
            "CidrIp": { "Ref": "VPCCidr" }
        }, {
            "IpProtocol": "tcp", "FromPort": "443", "ToPort": "443",
            "CidrIp": { "Ref": "VPCCidr" }
        }],
        "SecurityGroupEgress": [{
            "IpProtocol": "-1",
            "CidrIp": { "Ref": "VPCCidr" }
        }]
    }
}

There are only 2 important aspects to the PrivateApiLoadBalancer. First, it is attached to the same subnets as our ECS task definition. Secondly, it has a security group configured which allows for incoming HTTP/HTTPS traffic and outgoing traffic to any IP in the VPC CIDR address. The egress rule is required for the load balancer to be able to check the health of its targets.

For the load balancer to perform some actions we need to configure listeners that define its behavior in response to incoming traffic.

"PrivateApiLoadBalancerHttpListener": {
    "Type": "AWS::ElasticLoadBalancingV2::Listener",
    "Properties": {
        "Port": 80,
        "Protocol": "HTTP",
        "LoadBalancerArn": { "Ref": "PrivateApiLoadBalancer" },
        "DefaultActions": [{
            "Type": "forward",
            "TargetGroupArn": {
                "Ref": "PrivateApiLoadBalancerInvalidHostGroup"
            }
        }]
    }
},
"PrivateApiLoadBalancerInvalidHostGroup": {
    "Type": "AWS::ElasticLoadBalancingV2::TargetGroup",
    "Properties": {
        "Protocol": "HTTP",
        "Port": 80,
        "VpcId": { "Ref": "VPC" },
        "Name": "invalid-target-group"
    }
}

The PrivateApiLoadBalancerHttpListener specifies that the HTTP request to PrivateApiLoadBalancer on port 80 should be routed to PrivateApiLoadBalancerInvalidHostGroup. The AWS::ElasticLoadBalancingV2::Listener requires us to set the DefaultActions. The above setup allows us to fail in a consistent manner. Unless there are AWS::ElasticLoadBalancingV2::ListenerRule which match an incoming the HTTP request it will be routed to an empty AWS::ElasticLoadBalancingV2::TargetGroup which in turn will result in 502 Bad Gateway. This allows us to effectively decouple the DefaultActions of the load balancer listener from a specific ECS service instance.

Attach ECS service to an Application Load Balancer

It is time to route a request from the Application Load Balancer to the ECS service instances. The AWS::ElasticLoadBalancingV2::ListenerRule allows us to configure such behavior:

"ServiceAAlbHttpListenerRule": {
    "Type": "AWS::ElasticLoadBalancingV2::ListenerRule",
    "Properties": {
        "Actions": [{
            "Type": "forward",
            "TargetGroupArn": { "Ref": "ServiceAAlbTargetGroup" }
        }],
        "Priority": 1,
        "Conditions": [{
            "Field": "host-header",
            "Values": [ "service-a.in.example.com"]
        }],
        "ListenerArn": {
            "Ref": "PrivateApiLoadBalancerHttpListener"
        }
    }
},
"ServiceAAlbTargetGroup": {
    "Type": "AWS::ElasticLoadBalancingV2::TargetGroup",
    "Properties": {
        "Protocol": "HTTP",
        "Port": 8080,
        "HealthCheckPath": "/application-status/health",
        "HealthyThresholdCount": 2,
        "UnhealthyThresholdCount": 10,
        "HealthCheckIntervalSeconds": 5,
        "HealthCheckTimeoutSeconds": 4,
        "VpcId": { "Ref": "VPC" },
        "Name": "service-a-target-group"
    }
}

The ServiceAAlbHttpListenerRule states that any request with a Host equal to service-a.in.example.com should be routed to an instance from the ServiceAAlbTargetGroup. The ServiceAAlbTargetGroup group is referenced from the ServiceA definition and contains all running and healthy instances of our task. This means that in order to call the HTTP API exposed by ServiceA we will simply use the service-a.in.example.com domain name.

A private DNS record

The last part is to define a Private Hosted Zone and a DNS Record Set so that a DNS look up, happening inside the VPC, for service-a.in.example.com results in an IP address of the PrivateApiLoadBalancer.

"PrivateHostedZone": {
    "Type": "AWS::Route53::HostedZone",
    "Properties": {
        "VPCs": [{
            "VPCId": { "Ref": "VPC" },
            "VPCRegion": { "Ref": "AWS::Region" }
        }],
        "Name": "in.example.com"
    }
},
 "ServiceALoadBalancerDNSRecord": {
    "Type": "AWS::Route53::RecordSet",
    "Properties": {
        "Name": "service-a.in.example.com",
        "HostedZoneId": { "Ref": "PrivateHostedZone" },
        "ResourceRecords": [{
            "Fn::GetAtt": ["PrivateApiLoadBalancer", "DNSName"]
        }],
        "TTL": 60,
        "Type": "CNAME"
    }
}

The PrivateHostedZone merely defines the name of the domain which we will use for addressing ECS services running inside our VPC. It is recommended to use a domain name that you control so that a mistake in the DNS configuration will not cause your services to call an address that you do not own. The ServiceALoadBalancerDNSRecord defines a CNAME that uses PrivateApiLoadBalancer AWS assigned DNS name. This way the lookup for service-a.in.example.com will effectively resolve to multiple IPs in different availability zones.

With the above configuration in place we can call an HTTP API exposed privately by a load balanced service running inside VPC on ECS cluster using the most natural way i.e. a DNS name.

How to deploy a service to Amazon Elastic Container Service with CloudFormation?

Mon, 19 Feb 2018 00:00:00 GMT

Containers are becoming the standard way of deploying software. Every cloud vendor now offers one or multiple ways to run containers on their platform. Most of our clients uses AWS to host their SaaS solution. As part of a new development for one of our clients we have decided to move away from Elastic Beanstalk and embrace containers. Amazon Elastic Container Service is an orchestration service that supports Docker containers and is generally available for over a year. Given our small development team it seemed like the best choice since it takes away most of the cluster management headaches. In this post I will describe how we deploy a container to ECS using CloudFormation.

ECS Cluster definition

At Bright Inventions we often use CloudFormation for infrastructure configuration since it allows us to version and track changes easily. The first piece of infrastructure we need is an ECS cluster. A cluster is a logical group of tasks/containers running inside ECS. In particular, since we will be using EC2 Launch Type, a cluster can also be though of as a group of EC2 instances with an ECS Agent installed.

"ECSMainCluster": {
    "Type": "AWS::ECS::Cluster",
    "Properties": {
        "ClusterName": "app-stack-main"
    }
},
"ECSAutoScalingGroup": {
    "Type": "AWS::AutoScaling::AutoScalingGroup",
    "Properties": {
        "VPCZoneIdentifier": [
            { "Ref": "PrivateASubnet" },
            { "Ref": "PrivateBSubnet" }
        ],
        "LaunchConfigurationName": {
            "Ref": "ContainerHostInstances"
        },
        "MinSize": "1",
        "MaxSize": "6",
        "DesiredCapacity": "1",
        "Tags": [
            {
                "Key": "Name",
                "Value": "app-stack-ecs",
                "PropagateAtLaunch": true
            }
        ]
    },
    "CreationPolicy": {
        "ResourceSignal": {
            "Timeout": "PT5M"
        }
    },
    "UpdatePolicy": {
        "AutoScalingReplacingUpdate": {
            "WillReplace": "true"
        }
    }
}

As you can see above, the ECSMainCluster is mostly a declaration. What follows is an auto scaling group that will launch and manage EC2 instances. The VPCZoneIdentifier lists 2 VPC subnets created in separate availability zones. This is vital for availability as it causes the EC2 instances to run on physically separate hardware. For brevity, I’ve omitted their configuration from this post. However, if you are interested in this topic head over to this post. The specified LaunchConfigurationName named ContainerHostInstances details how the EC2 instance should look like.

"ContainerHostInstances": {
    "Type": "AWS::AutoScaling::LaunchConfiguration",
    "Properties": {
        "ImageId": "ami-880d64f1",
        "SecurityGroups": [
            { "Ref": "ECSSecurityGroup" }
        ],
        "InstanceType": "t2.medium",
        "IamInstanceProfile": { "Ref": "ECSHostEC2InstanceProfile" },
        "KeyName": "private-key-pair",
        "UserData": {
            "Fn::Base64": {
                "Fn::Join": [
                    "",
                    [
                        "#!/bin/bash -xe\n",
                        "echo ECS_CLUSTER=",
                        {
                            "Ref": "ECSMainCluster"
                        },
                        " >> /etc/ecs/ecs.config\n",
                        "yum install -y aws-cfn-bootstrap\n",
                        "/opt/aws/bin/cfn-signal -e $? ",
                        "         --stack ",
                        {
                            "Ref": "AWS::StackName"
                        },
                        "         --resource ECSAutoScalingGroup ",
                        "         --region ",
                        {
                            "Ref": "AWS::Region"
                        },
                        "\n"
                    ]
                ]
            }
        }
    }
},
"ECSHostEC2InstanceProfile": {
    "Type": "AWS::IAM::InstanceProfile",
    "Properties": {
        "Path": "/",
        "Roles": [
            {
                "Ref": "ECSHostEC2Role"
            }
        ]
    }
},

The first important property is the ImageId which uses Amazon ECS-optimized Linux AMI ID. Next we have a security group that adds rules for incoming traffic on application ports. Next we have IamInstanceProfile which references an instance profile ECSHostEC2InstanceProfile that in turn assumes a role policy required by the ECS Agent to deploy and configure containers. Inside UserData we define a shell script that informs ECS Agent about the cluster it is running in. I will omit ECSHostEC2Role definition since it is well described in the documentation.

With the above we are now ready to deploy an ECS Cluster through CloudFormation template. However, a cluster without containers is pretty meaningless.

ECS Service and Task definition

In AWS lingo an ECS Service describes a minimal configuration required to deploy and run a Task Definition. A Task Definition in turn describes how to configure and run a set of containers that form a single logical component.

"EmailSenderService": {
    "Type": "AWS::ECS::Service",
    "Properties": {
        "Cluster": { "Ref": "ECSMainCluster" },
        "DesiredCount": 2,
        "DeploymentConfiguration": { "MinimumHealthyPercent": 50 },
        "Role": { "Ref": "ECSServiceRole" },
        "TaskDefinition": { "Ref": "EmailSenderTask" }
    }
},
"EmailSenderTask": {
    "Type": "AWS::ECS::TaskDefinition",
    "Properties": {
        "Family": "app-stack-email-sender",
        "ContainerDefinitions": [{
            "Name": "app-stack-email-sender",
            "Essential": "true",
            "Image": { "Ref": "EmailSenderTaskDockerImage" },
            "LogConfiguration": {
                "LogDriver": "awslogs",
                "Options": {
                    "awslogs-group": { "Ref": "EmailSenderLogsGroup" },
                    "awslogs-region": { "Ref": "AWS::Region" },
                    "awslogs-stream-prefix": "email-sender",
                    "awslogs-datetime-format": "%Y-%m-%d %H:%M:%S.%L"
                }
            },
            "PortMappings": [{ "ContainerPort": 8080 }],
            "Environment": [{
                "Name": "DEPLOY_ENV",
                "Value": { "Ref": "DeployEnv" }
            }]
        }]
    }
},
"EmailSenderLogsGroup": {
    "Type": "AWS::Logs::LogGroup",
    "Properties": {
        "LogGroupName": "app-stack-email-sender",
        "RetentionInDays": 14
    }
}

The EmailSenderService is pretty straightforward to understand. The EmailSenderTask defines a single container. The app-stack-email-sender task definition states that the Image is a reference to a parameter passed in to the CloudFormation template when creating or updating the stack. Its value must be a name of a Docker image that can be pulled by the ECS Agent. The repository can either be public or private. When hosting your own, private Docker image repository you need to make sure the ECS Agent has the correct credentials configured. Thankfully there is Elastic Container Registry which offers private repositories that are automatically configured when using ECS as long as ECSHostEC2Role policy allows ECR related actions.

Next we have the LogConfiguration that pushes containers logs to the EmailSenderLogsGroup CloudWatch Log Group so that we can can inspect them through AWS Console. The PortMappings lists ports exposed by a running container. Note that we have not defined the host port and it will get assigned automatically. This is important when running multiple instances of the same container. I’ll describe it in a bit more detail in the next post. Last but not least the Environment section lists environment variables passed to the container instances on startup. Here we are referencing a DeployEnv stack parameter that allows us to inform the application running inside the container about the current deployment environment e.g. staging vs production.

As you can see above it takes couple of steps to use CloudFormation to deploy a container to ECS. It is true that it requires more configuration than Elastic Beanstalk. However, it allows for better utilization of EC2 instances and an uniform approach to deployment and configuration regardless of the application technology used inside the container. Moreover it is more future proof as with few adjustments it should be possible to switch to Fargate Launch Mode. Using this mode releases us from the burden of EC2 ECS cluster management tasks. Deploying more services and tasks will require separate CloudFormation resource definitions. However, with the help of cloudform it easy keep the CloudFormation template DRY.

In-memory database tests with Querydsl

Mon, 12 Feb 2018 22:14:00 GMT

Writing tests is an important skill of a software engineer. I used to write lots of very focused, narrow unit tests. However, I often found such tests to hinder refactoring and barely help in catching regressions. Whether such issues were caused by my poor design choices or are intrinsic to unit tests is not the focus of this post. However, the fact is that nowadays I tend to write more coarse-grained, integration style tests. There is one downside to such approach: speed. For instance, using Hibernate with a full fledged database is relatively slow compared to using a fake repository implementation. Today I write about abstracting the database access using Querydsl in a way that aids testing.

Querydsl is awesome

Querydsl is a set of libraries that, as the name implies, provides strongly typed Domain Specific Language to execute queries. Querydsl supports many data access technologies e.g. JDBC, Hibernate, JDO. The following example in Kotlin illustrates how a DSL generated based on entity class can be used to find some entities through JPA interface:

val queryFactory: JPAQueryFactory = ...
val userEmailToSearch = "alamakota@gmail.com"
val user = queryFactory.query()
    .from(QUser.user)
    .where(QUser.user.email.eq(userEmailToSearch))
    .select(QUser.user)
    .fetchOne()

One important option available is the Collections module that offers an integration to POJO collections and beans. The following example in Kotlin shows how a list of users can be queried:

val users = listOf(userAlan, userBob, userAlice)
val user = CollQuery<Nothing>()
    .from(QUser.user, users)
    .where(QUser.user.email.eq(userEmailToSearch))
    .select(QUser.user)
    .fetchOne()

Abstract the complex away

The above examples look similar thanks to common interface provided by Querydsl. However, while the default DSL is very capable I found it a bit verbose in the most common cases. For that matter let us define a bit simpler interface that will allow for finding entities given some criteria

interface EntityQueries {
    fun <TQEntity : EntityPath<TEntity>, TEntity : Any> findFirst(
        qEntity: TQEntity, 
        where: (TQEntity) -> Predicate? = { null }, 
        orderBy: ((TQEntity) -> OrderSpecifier<*>?) = { null }): TEntity? 
}

val queries:EntityQueries = ....

val ala = queries.findFirst(QUser.user, where = { it.email.eq("ala@gmail.com") })
val latestUser = queries.findFirst(QUser.user, orderBy = { it.created.desc() })

The above interface allows us to express commonly used queries in a more succinct fashion.

Define production implementation

With Querydsl it is easy enough to implement the EntityQueries interface. First the production implementation delegating to JPA for actual data access technology:

class QueryDslDomainQueryFactory(private val queryFactory: JPAQueryFactory) : EntityQueries {
    override fun <TQEntity : EntityPath<TEntity>, TEntity : Any> findFirst(qEntity: TQEntity, where: (TQEntity) -> Predicate?, orderBy: (TQEntity) -> OrderSpecifier<*>?): TEntity? {
        return queryFactory.query()
            .from(qEntity)
            .where(where(qEntity))
            .apply { orderBy(qEntity)?.let { this.orderBy(it) }  }
            .select(qEntity)
            .fetchFirst()
    }
}

The above lets us use the EntityQueries interface instead of JPA in e.g. Spring controllers like so:

@RestController
class UsersController(private val queries: EntityQueries) {
    @GetMapping("/users")
    fun getByEmail(@RequestParam email: String) = queries.findFirst(QUser.user, where = { it.email.eq(email) })
}

One of the Spring recommended ways to abstract the specifics of query technology is to use repository interfaces e.g:

interface UserRepository : Repository<User, Long> {
  fun findByEmail(String email): User?
}

Such interface would be magically implemented by Spring runtime and put in the application context. The approach may seem appealing at first since we do not have to implement the interface. There are however, multiple issues:

an application context is required which in turn is slow to bootstrap
there is no compile time checks
the refactoring is harder without a special support from IDE
the actual behavior is hard to figure out without a careful documentation lecture (what will happen if e.g. there are multiple users with the same email?)

The EntityQueries invocation to find users by email is almost as readable as findByEmail but does not suffer from any of downsides listed above. Encapsulating more complex filtering logic can be done with a simple extension method or a more elaborate Specification pattern.

Using in-memory database in tests

We can use Spring test helpers to ease writing tests involving an application context that lets us inject e.g. UsersController instance to invoke its methods. However, such tests are, comparatively, very slow to run and thus cause the feedback loop to be much slower. Fortunately the EntityQueries abstraction is very easy to implement using POJO in-memory collections.

class InMemoryEntityQueries : QueriesBase(), EntityQueries {
    val entities = mutableMapOf<Class<*>, MutableList<*>>()

    override fun <TQEntity : EntityPath<TEntity>, TEntity : Any> findFirst(qEntity: TQEntity, where: (TQEntity) -> Predicate?, orderBy: (TQEntity) -> OrderSpecifier<*>?): TEntity? {
        val entities = entities.getOrPut(qEntity.type, { mutableListOf<TEntity>() }) as List<TEntity>
        return CollQuery<Nothing>()
            .from(qEntity, entities)
            .where(where(qEntity))
            .apply { orderBy(qEntity)?.let { this.orderBy(it) } }
            .select(qEntity)
            .fetchFirst()
    }
}

The above implementation looks almost exactly the same as the production one. We can of course try to extract the common code to make things more DRY. However, the most important observation is that we delegate to Querydsl implementation for the important filtering and ordering logic. This can increase our confidence that the fake implementation behaves the same as production one with only difference being the actual entity storage.

Given the above implementation we can now easily replace the UsersController dependency and instantiate it as a regular POJO:

class UsersControllerTests {
    val db = InMemoryEntityQueries()
    val controller = UsersController(db)

    @Test
    fun canFindByEmail(){
        db.entities[User::class.java] =  listOf(User(email = "ala@gmail.com"), User(email = "ola@gmail.com"))

        controller.getByEmail("ola@gmail.com").email.shouldEqual("ola@gmail.com")
        controller.getByEmail("peter@gmail.com").shouldEqual(null)
    }
}

Notes on in-memory implementation

The EntityQueries interface above is obviously a simplified version. The most important missing piece is the ability to save entities. However, this is not a hard thing to implement given the in-memory implementation. We can, for instance, make use of the fact that all of our entities are marked JPA Persistence annotations to find a field marked with @Id, generate the id and assign it based on the contents of the entities variable. Another approach is to mark all entities with a dedicated interface e.g.

interface HasId<TId> {
    var id: TId
}

An entity implementing HasId could be checked in the save method of the in-memory implementation and assigned with a unique identifier e.g.:

fun <TEntity: HasId<Long>> save(entity: TEntity) {
    val entities = entities.getOrPut(entity.javaClass, { mutableListOf<TEntity>() }) as List<TEntity>
    if(entity.id == null){
        entity.id = (entities.map { it.id }.max() ?: 0) + 1
    }
    entities += entity
}

Following the above approach we can easily add missing operations e.g. to remove an entity and that in turn allows us to write even more tests that run fully in-memory. It is worth noting that using an in-memory database implementation works best for queries that fetch, save or update one or multiple instances. As soon as we need to use a features natural to a database technology e.g. joins in SQL, we are better of connecting to a real database. While Querydsl collections module supports both join and group operations the in-memory implementation is often not equivalent to the database one especially around null values handling.

How to publish a library to a Maven repository with the maven-publish plugin

Tue, 06 Feb 2018 22:14:00 GMT

A seasoned developer now and then creates a piece of code that he or she would like to reuse in a different project. When such time comes it is useful to know how to publish a library so that it can easily be incorporated into a different project. In this post I will describe how to publish a Kotlin library to JCenter with maven-publish and com.jfrog.bintray Gradle plugins.

Gradle Maven plugins

The first step is to apply Maven plugin. The plugin adds support for deploying artifacts to Maven repositories. Note that in case of multi-project build e.g. ShouldKO the Maven plugin should be applied to every project that defines some artifact to be published. You can use allprojects to get rid of duplication e.g.:

allprojects {
    repositories {
        jcenter()
    }

    apply plugin: 'kotlin'
    apply plugin: 'maven'

    group "pl.miensol.shouldko"
}

For the com.jfrog.bintray plugin, used later on, to work nicely with Maven artifacts we need to apply additional Gradle plugin. This additional piece is the maven-publish plugin which provides ability to publish artifacts in Maven format. All we need to do is to apply plugin: 'maven-publish' in the main project.

Define Maven publishing

The com.jfrog.bintray plugin relies on properly defined Maven Publications. The Gradle DSL allows us to define them easily basing on project properties e.g.

publishing {
    publications {
        hamcrest(MavenPublication) {
            def project = project(':hamcrest')
            from project.components.java
            artifact project.sourcesJar { // not required, includes sourcesJar with correct classifer
                classifier "sources"
            }
            groupId group
            artifactId project.name
            version project.version
        }

        core(MavenPublication) {
            def project = project(':core')
            from project.components.java
            artifact project.sourcesJar {
                classifier "sources"
            }
            groupId group
            artifactId project.name
            version project.version
        }
    }
}

The above Maven Publications include sources artifact. Publishing additional classifiers for artifacts is important since it allows for IDE to show a documentation popup or debug through the library source code. However, one needs to define it first as it is not included by default when applying java or kotlin plugins to a Gradle project. This is easily done as follows:

allprojects {
    task sourcesJar(type: Jar, dependsOn: classes) {
        from sourceSets.main.allSource
    }
}

Project versioning

As you saw above, we have used project.version to indicate a version to MavenPublication. There are multiple strategies to version software but the Semantic Versioning scheme is widely accepted as a standard when it comes to libraries. If you wish to use it then there are plugins available for Gradle to simplify the mundane tasks of maintaining pre-release and patch versions. I like the set of plugin from ajoberstar that provide an opinionated way to version your project based on git tags. Applying them is easy:

plugins {
    id "org.ajoberstar.grgit" version "1.7.2"
    id "org.ajoberstar.release-opinion" version "1.7.2"
}

Now when you issue e.g. gradle build the plugin will infer a next version based on your git repository state:

> Configure project : 
Inferred project: shouldko, version: 0.1.5-dev.0.uncommitted+4f71d34

Bintray upload

Finally, when we are ready to upload our library and make it available for everyone we need to set up a Bintray account. Once we have it, on the profile page we can access API key required to configure the Bintray Gradle plugin.

bintray {
    user = project.hasProperty('bintrayUser') ? project.property('bintrayUser') : System.getenv('BINTRAY_USER')
    key = project.hasProperty('bintrayApiKey') ? project.property('bintrayApiKey') : System.getenv('BINTRAY_API_KEY')
    publications = ['core', 'hamcrest']
    pkg {
        repo = 'maven'
        name = 'shouldko'
        desc = 'Adds source line to tests assertion messages'
        userOrg = 'brightinventions'
        licenses = ['MIT']
        vcsUrl = 'https://github.com/bright/shouldko.git'
        labels = ['tests', 'hamcrest', 'junit']
    }
}

The Bintray API key should be kept private and by no means included in the source code repository. We can configure user and key by looking at project properties and if not available using environment variables. This way there is no need to expose them publicly.

gradle build bintrayUpload -PbintrayUser=<apiUser> -PbintrayApiKey=<apikKey>

The repo is the name of the Bintray repository. You can use the same Bintray repository to host multiple projects.

The Bintray plugin is very taciturn thus I like to add some log message to see when the bintrayUpload completes:

afterEvaluate {
    tasks.bintrayUpload.doLast {
        logger.lifecycle("Uploaded artifacts to bintray at version $version")
    }
}

Travis build

Every project should have at least some form of continuous integration. For open source software there are at least couple of free build servers available. Travis is probably the most popular one. For gradle project Travis will by default call build. If you would like to upload the build artifacts to Bintray whenever successful build completes you need to add a line to script section of the .travis.yml like so:

script:
  - ./gradlew build
  - ./gradlew bintrayUpload

Obviously the Bintray credentials need to be configured as well which can be done through a project configuration page:

Now, the Gradle git plugin will create a development version and publish it to Bintray on every Travis build.

Tag to release

Whenever you want to release a new version of the library you now can simply tag a particular version e.g.

git tag 0.1.4
git push origin 0.1.4

After a local or continuos integration build completes you should see a new version in the Bintray web application. From there you need to publish the version.

Use the new library

Once a version is published, you can consume it from a maven or gradle project easily. Until you link your package to JCenter, you need to inform your build system about a new maven repository location e.g.:

repositories {
    jcenter()
    maven { url 'https://dl.bintray.com/brightinventions/maven' }
}

Note that brightinventions is the organization user name and maven is the repository name mentioned above. You are now, finally able to consume your library 🎉:

compile 'pl.miensol.shouldko:hamcrest:0.1.4'

Enjoy!

Use ShouldKO for better assertion messages

Wed, 17 Jan 2018 00:00:00 GMT

Most of us write tests these days. Whether they are unit, integration, end-to-end or performance tests once written we often do not go back to them until they fail. It is thus vital to have a clear assertion message when a test fails.

Hamcrest

I think Hamcrest is the most popular assertion library available in Java and Kotlin ecosystem. Let us look at an oversimplified example of Money class:

import java.math.BigDecimal
import java.util.*

data class Money(val amount: BigDecimal, val currency: Currency) {
    operator fun plus(other: Money): Money {
        if (currency != other.currency) {
            throw IllegalArgumentException("Cannot add $this to $other. Currencies must match.")
        }

        // an accidentally introduced bug 😈
        val newAmount = amount + other.amount + 10.0.toBigDecimal()
        return copy(amount = newAmount)
    }
}

Imagine we write a test for the plus operator using JUnit and Hamcrest:

class MoneyTests {
    val usd = Currency.getInstance("USD")

    @Test
    fun can_add() {
        val usd100 = Money(100.toBigDecimal(), usd)
        val usd50 = Money(50.toBigDecimal(), usd)

        assertThat((usd100 + usd50).amount, equalTo(150.toBigDecimal()))
    }
}

The test will fail with the following message:

java.lang.AssertionError: 
Expected: <150>
     but: was <160>

Let’s see how we can improve on that.

ShouldKO: better assertion messages for Kotlin

ShouldKO is a simple library I’ve come up with that improves the assertion messages. Its idea is based on assertion libraries available in .NET e.g. Shouldly. Let us see how the tests looks like using ShouldKO:

class MoneyTests {
    val usd = Currency.getInstance("USD")

    @Test
    fun can_add() {
        val usd100 = Money(100.toBigDecimal(), usd)
        val usd50 = Money(50.toBigDecimal(), usd)

        (usd100 + usd50).amount.shouldEqual(150.toBigDecimal())
    }
}

In my opinion ShouldKO’s assertion syntax improves readability. However, this is not where ShouldKO main improvement is. Follow the improved assertion message:

java.lang.AssertionError: (usd100 + usd50).amount 
Expected: <150>
     but: was <160>

ShouldKO incorporates a source code line with the assertion into the assertion message itself. This comes really handy when we have multiple lines with assertions that form one logical condition. This is a small thing, but can greatly improve debugging test issues especially when all we have is a log file produced by a test run.

Installation of ShouldKO

ShouldKO is currently available on Jitpack. You need to first add Jitpack to your repositories:

repositories {
    maven { url 'https://jitpack.io' }
    mavenCentral()
}

And include the library in your tests e.g.:

testImplementation 'com.github.miensol.shouldko:hamcrest:v0.1.0'

ShouldKO’s Hamcrest library allows for using any Hamcrest matcher.

Multi tenancy task scheduler

Thu, 04 Jan 2018 22:14:00 GMT

Last time I showed how to extend Spring default request handler adapter so that we are able to schedule or reject incoming requests. The goal of the TenantTaskCoordinator is to:

queue requests for processing
limit the maximum number of concurrently processed requests
reject requests after the maximum queue size is reached
interrupt processing of a request upon an upstream subscription disposal

Tenant task coordinator execute method

Our entry point into TenantTaskCoordinator is a single method fun <T : Any> execute(job: Callable<T>): Mono<T>:

    fun <T : Any> execute(job: Callable<T>): Mono<T> {
        return Mono.create({ outsideSink ->
            val _workInProgressWasDecremented = AtomicBoolean(false)
            fun decrementOnce() {
                if (_workInProgressWasDecremented.compareAndSet(false, true)) {
                    currentWorkInProgressCounter.decrementAndGet()
                }
            }

            val workInProgress = currentWorkInProgressCounter.incrementAndGet()
            if (workInProgress > maximumWorkInProgress) {
                outsideSink.error(TooManyTasks("Work in progress $workInProgress exceeds $maximumWorkInProgress jobs in $name"))
                decrementOnce()
            } else {
                val singleJob = Mono.fromCallable(job).doAfterTerminate {
                    decrementOnce()
                }

                val delayedTask = Task(name, singleJob as Mono<Any>, outsideSink as MonoSink<Any>)

                outsideSink.onCancel {
                    delayedTask.outsideCancel()
                    decrementOnce()
                }
                
                taskSink.next(delayedTask)
            }
        })
    }

The first step is to return Mono<T> which is simply done with Mono.create. The sink we get passed is used to control the outcome observed from outside. It also allows for registering an onCancel callback invoked when the upstream cancels its subscription.

The _workInProgressWasDecremented is used to guard and decrement the currentWorkInProgressCounter in a thread safe fashion. We first check whether we have immediately exceeded the maximum number of queued jobs. If the threshold is reached, we notify the observer about the error with outsideSink.error.

If we have enough capacity to a perform job, we convert it to a reactive world with Mono.fromCallable and attach a doAfterTerminate callback that decrements the work in progress counter. The Task class links singleJob and outsideSink so that they are both accessible while processing. Finally, we schedule the task through taskSink.next(delayedTask).

Task coordinator state

Let’s have a look at the task coordinator state variables and how they are initialized:

class TenantTaskCoordinator(private val scheduler: Scheduler,
                            val maximumConcurrency: Int = 1,
                            val maximumQueueSize: Int = 50,
                            val name: String = "") : AutoCloseable {

    private val maximumWorkInProgress = maximumQueueSize + maximumConcurrency

    private val maxBufferSize = maximumWorkInProgress * 2

    val currentWorkInProgressCounter = AtomicInteger()

    private lateinit var taskSink: FluxSink<Task>

    private val taskSource = Flux.create<Task>({ taskSink = it }, FluxSink.OverflowStrategy.BUFFER)

    private val processSinkOnErrorResume = processSinkWithLimitedConcurrency()
        .onErrorResume { error: Throwable? ->
            LOG.warn("name={} Error processing sink with limited concurrency", name, error)
            processSinkWithLimitedConcurrency()
        }

The first interesting part is how we setup taskSink by using Flux.create. For clarity, we explicitly pass FluxSink.OverflowStrategy.BUFFER so that tasks are buffered in case they outpace the processor. The name is used to get better log messages. Finally, we call processSinkWithLimitedConcurrency to start task processing using the given scheduler. Interestingly the onErrorResume restarts the processing in case we have a bug.

Task coordinator concurrent processing

The most important and tricky to figure out part is to correctly process jobs. It took me several back and forth steps until I got the order of reactive API calls right.

    private fun processSinkWithLimitedConcurrency(): Flux<Any> {
        return taskSource
            .filter { !it.isCancelled }
            .flatMap({ task ->
                task.work
                    .doOnError(task::onError)
                    .doOnSuccess(task::onSuccess)
                    .subscribeOn(scheduler)
                    .timeout(task.outsideTimeout)
                    .onErrorReturn(task)
            }, maximumConcurrency, maxBufferSize)
    }

First, we filter out tasks that are already cancelled. Then, we use flatMap overload to process tasks with given maximum concurrency. The flatMap callback delegates most of the work to the mentioned Task instance. The onErrorReturn effectively suppresses any errors that might occur during task execution. Let’s see how the inner Task class looks like:

private data class Task(val name: String,
                            private val job: Mono<Any>,
                            val outsideSink: MonoSink<Any>,
                            @field:Volatile var isCancelled: Boolean = false) {

        val work: Mono<Any> get() = if (isCancelled) Mono.empty() else job

        lateinit var outsideTimeoutSink: MonoSink<Task>
        val outsideTimeout = Mono.create<Task> { outsideTimeoutSink = it }

        fun outsideCancel() {
            isCancelled = true
            outsideTimeoutSink.success(this)
        }

        fun onSuccess(result: Any?) {
            outsideSink.success(result)
        }
        
        fun onError(error: Throwable) {
            LOG.warn("Task.onError {}", this, error)
            outsideSink.error(error)
        }
    }

The job argument is the Callable passed to the execute method. The outsideTimeout signals when the task instance subscription is cancelled. The signal is propagated inside processSinkWithLimitedConcurrency with a Mono.timeout call and breaks the task processing. Last but not least the onSuccess and onError simply push the result or error to the outsideSink effectively notifying the observer of the result.

The TenantTaskCoordinator was not simple to figure out given the requirements mentioned at the begging of the post. I’m pleased with the final result although I must say it was not intuitive to figure out how to combine all the nuts and bolts of Reactor library to achieve the desired outcome.

Multi tenancy in Spring MVC

Tue, 12 Dec 2017 22:14:00 GMT

One of our clients aimed to replace old, often DOS based, point of sale systems with a cloud based, SaaS modeled solution. [At Bright Inventions]({{ site.url }}) we have developed all required components including AWS based back-end processing requests originating from multiple clients. Each business that uses the SaaS point of sale can be considered a tenant in a multi-tenant environment. There many aspects involved when developing multi-tenant application with data isolation and partitioning being the most discussed topic. However, today I would like to focus on computational and resource isolation aspect.

Controlled resource usage

In the discussed case each tenant would have multiple iOS based API clients. The exact number varies from 1 to couple of dozens. Each iOS application would be open constantly throughout a sales day and execute frequent requests against the back-end API. In the iOS application there was a code that polls the server for data changes in frequent and regular intervals. Unfortunately a bug slipped through a code review and caused the app to ask the server for changes around 50 times per second instead of once in half of a minute. The bug caused an explosion of API requests issued by a single API client with a throughput 10 to 100 times bigger than expected. To make things worse the rate at which the bug increased polling frequency exceeded back-end the scaling out policy. Soon all request processing threads were busy processing requests issued by only a small percentile of API clients.

In a multi-tenant application one needs to take special care to prevent tenant “A” from affecting, even indirectly, tenant “B” operations. We have failed that requirement on the CPU/thread pool level and that caused the support lines to be hot.

Reverse proxy request rate limit

The first solution that comes to mind is to apply a per API client request rate limiting. In fact this solution is so common that it is available as a configuration opt-in in many servers. For instance in NGINX you could do:

limit_req_zone $binary_remote_addr zone=mylimit:10m rate=10r/s;

server {
    location /login/ {
        limit_req zone=mylimit burst=20;

        proxy_pass http://my_upstream;
    }
}

The above would only allow up to 10 request per second from the same IP address. Any request that comes in at higher rate would be queued up to specified capacity (burst=20). Any request above the limit would get rejected with 503 status code.

The nginx approach is battle tested and fairly easy to apply. Instead of using IP address it would be better to group requests by a tenant identifier. However, it may not be easy to determine exactly which tenant is making the request unless the information is easily available in the request headers. For that matter it is good to consider sending the API client identification using a custom HTTP header. For instance if the API client provides X-Tenant-Id: tenant.1 you can use it as limit_req_zone $http_x_tenant_id zone=mylimit:10m rate=10r/s;. When using JWT, you often can determine who is making the request by parsing the Authorization header value.

Spring MVC request rate limit

It is often not feasible to apply the request rate limit at the reverse proxy level. In such scenario we can apply the limit inside Spring MVC application. For start one can try suing Servlet Filter. There are several solutions available including a DoSFilter that is part of Jetty project.

Using a ready-made Servlet Filter is often sufficient especially when the available customization options suit our needs. In case of our client however, we wanted the limits to depend on the size of the client. In other words the more service you buy, the more resources are available to you. Moreover, I wanted to have a have fine-grained control at a controller action level. To my surprise such behavior was not easy to accomplish using AsyncHandlerInterceptor. Fortunately I did find a way to achieve a desired result using a mix of extensibility points and hacks.

The first step is to customize RequestMappingHandlerAdapter used by Spring MVC to transform @RequestMapping annotation into handler classes. The following configuration class in Kotlin achieves just that:

@Configuration
class WebMvcConfiguration : DelegatingWebMvcConfiguration() {

    @Autowired(required = false)
    private val mvcProperties: WebMvcProperties? = null

    @Inject lateinit var reactiveRequestCommandFactory: ReactiveRequestCommandFactory
    @Inject lateinit var reactiveRequestsProperties: ReactiveRequestsConfiguration.RequestsProperties

    @Bean
    override fun requestMappingHandlerAdapter(): RequestMappingHandlerAdapter {
        //copy pasted from WebMvcConfigurationSupport
        val argumentResolvers = ArrayList<HandlerMethodArgumentResolver>()
        addArgumentResolvers(argumentResolvers)

        val returnValueHandlers = ArrayList<HandlerMethodReturnValueHandler>()
        addReturnValueHandlers(returnValueHandlers)

        val adapter = RateLimitingRequestMappingHandlerAdapter(reactiveRequestCommandFactory, reactiveRequestsProperties)

        adapter.setContentNegotiationManager(mvcContentNegotiationManager())
        adapter.messageConverters = messageConverters
        adapter.webBindingInitializer = configurableWebBindingInitializer
        adapter.customArgumentResolvers = argumentResolvers

        adapter.customReturnValueHandlers = returnValueHandlers

        val requestBodyAdvices = ArrayList<RequestBodyAdvice>()
        requestBodyAdvices.add(JsonViewRequestBodyAdvice())
        adapter.setRequestBodyAdvice(requestBodyAdvices)

        val responseBodyAdvices = ArrayList<ResponseBodyAdvice<*>>()
        responseBodyAdvices.add(JsonViewResponseBodyAdvice())
        adapter.setResponseBodyAdvice(responseBodyAdvices)

        configureAsync(adapter)


        adapter.setIgnoreDefaultModelOnRedirect(mvcProperties?.isIgnoreDefaultModelOnRedirect != false)
        return adapter
    }

    private fun configureAsync(adapter: RequestMappingHandlerAdapter) {
        //expose field publicly
        val configurer = object : AsyncSupportConfigurer() {
            public override fun getTaskExecutor() = super.getTaskExecutor()
            public override fun getTimeout() = super.getTimeout()
            public override fun getCallableInterceptors() = super.getCallableInterceptors()
            public override fun getDeferredResultInterceptors() = super.getDeferredResultInterceptors()
        }

        configureAsyncSupport(configurer)

        if (configurer.taskExecutor != null) {
            adapter.setTaskExecutor(configurer.taskExecutor)
        }

        if (configurer.timeout != null) {
            adapter.setAsyncRequestTimeout(configurer.timeout!!)
        }

        adapter.setCallableInterceptors(configurer.callableInterceptors)
        adapter.setDeferredResultInterceptors(configurer.deferredResultInterceptors)
    }
}

Note that we are injecting reactiveRequestCommandFactory and reactiveRequestsProperties and pass them into our core RateLimitingRequestMappingHandlerAdapter. All other code is a mostly a copy-paste from DelegatingWebMvcConfiguration base class.

@Target(AnnotationTarget.CLASS, AnnotationTarget.FUNCTION)
annotation class RequestCommand(
    val enabled: Boolean = true,
    val timeoutInMillis: Int = 60_000
)

class RateLimitingRequestMappingHandlerAdapter(private val reactiveRequestCommandFactory: ReactiveRequestCommandFactory,
                                               private val reactiveRequestProperties: ReactiveRequestsConfiguration.RequestsProperties) : RequestMappingHandlerAdapter() {
    private val handlerMethodConfigurationsCache = ConcurrentHashMap<HandlerMethod, RequestCommandConfiguration>()

    override fun createInvocableHandlerMethod(handlerMethod: HandlerMethod): ServletInvocableHandlerMethod? {
        val configuration = requestCommandConfigurationFor(handlerMethod)

        return when {
            configuration.enabled && reactiveRequestProperties.enabled -> CommandInvocableHandlerMethod(handlerMethod, reactiveRequestCommandFactory, configuration)
            else -> super.createInvocableHandlerMethod(handlerMethod)
        }
    }

    private fun requestCommandConfigurationFor(handlerMethod: HandlerMethod): RequestCommandConfiguration {
        return handlerMethodConfigurationsCache.getOrPut(handlerMethod) {
            val method = handlerMethod.getMethodAnnotation(RequestCommand::class.java)
            val methodOrController = method ?: AnnotatedElementUtils.findMergedAnnotation(handlerMethod.beanType, RequestCommand::class.java)
            methodOrController?.let { RequestCommandConfiguration(it) } ?: RequestCommandConfiguration.Default
        }
    }
}

Inside of createInvocableHandlerMethod we get the configuration for the handlerMethod determined by Spring MVC. The handlerMethod denotes a controller action. Then we decide if we should use the rate limiting handler or fallback to the default one. In case we need to apply rate limiting we switch the invocation to use custom CommandInvocableHandlerMethod:

class CommandInvocableHandlerMethod(private val handlerMethod: HandlerMethod,
                                    private val requestCommandFactory: RequestCommandFactory,
                                    private val configuration: RequestCommandConfiguration) : ServletInvocableHandlerMethod(handlerMethod) {
    private lateinit var returnValueHandlers: HandlerMethodReturnValueHandlerComposite

    override fun invokeForRequest(request: NativeWebRequest?, mavContainer: ModelAndViewContainer?, vararg providedArgs: Any?): Any {
        // same as super.invokeForRequest(request, mavContainer, *providedArgs)
        // but with request passed to do invoke
        val args = this.getMethodArgumentValuesCallable.invoke(request, mavContainer, providedArgs)
        val result = doInvokeWithRequest(request, args)
        return result
    }

    private fun doInvokeWithRequest(request: NativeWebRequest?, args: Array<out Any?>?): Any {
        val nativeRequest = request?.getNativeRequest(HttpServletRequest::class.java)

        // If the response has already set error status code tomcat will not wait for async result
        return if (nativeRequest != null && nativeRequest.dispatcherType == DispatcherType.REQUEST) {
            val callSuper = Callable {
                super.doInvoke(*(args ?: emptyArray()))
            }

            val job = callSuper

            val context = RequestCommandContext(configuration, handlerMethod, SecurityContextHolder.getContext(), job)

            val result = requestCommandFactory.createSingle(context)

            MonoDeferredResult(result)
        } else {
            super.doInvoke(*(args ?: emptyArray()))
        }
    }

    override fun setHandlerMethodReturnValueHandlers(returnValueHandlers: HandlerMethodReturnValueHandlerComposite?) {
        this.returnValueHandlers = returnValueHandlers!!
        super.setHandlerMethodReturnValueHandlers(returnValueHandlers)
    }

    override fun wrapConcurrentResult(result: Any?): ServletInvocableHandlerMethod {
        return ConcurrentResultHandlerMethod(result, ConcurrentResultMethodParameter(result))
    }

...

The above code is using private getMethodArgumentValues API to achieve the desired behavior‼ The doInvokeWithRequest checks if an asynchronous dispatch should be performed and if so creates a Mono that denotes the result of the controller action method invocation. RequestCommandContext stores the information about target controller action method and current security context. The security context needs to be preserved when invoking the controller action on a different thread. The ConcurrentResultHandlerMethod extends ServletInvocableHandlerMethod to add support for using Mono on regular, synchronous controller action. The core logic of rate limiting is delegated to ReactiveRequestCommandFactory:

interface ReactiveRequestCommandFactory {
    fun createSingle(context: RequestCommandContext): Mono<Optional<Any>>
}

The factory responsibilty it to convert a request context into an async result. Spring MVC 5 has built in support for Reactor hence we decided to use this implementation of Reactive Streams specification. The ReactiveRequestCommandFactory looks as follows:

@Component
class ReactorRequestCommandFactory(
    threadPoolPropertiesCalculator: ThreadPoolPropertiesCalculator,
    @param:Named("reactiveRequestsScheduler")
    private val reactiveRequestsScheduler: Schedule
) : ReactiveRequestCommandFactory {
    private val threadPoolPropertiesCalculator = HystrixConfigurationAwarePropertiesCalculator(threadPoolPropertiesCalculator)
    private val tenants = ConcurrentHashMap<String, TenantTaskCoordinator>()

    override fun createSingle(context: RequestCommandContext): Mono<Optional<Any>> {
        val properties = threadPoolPropertiesCalculator.newThreadPoolProperties(context)

        val taskCoordinator = tenants.computeIfAbsent(properties.threadPoolName) {
            TenantTaskCoordinator(reactiveRequestsScheduler,
                maximumConcurrency = properties.maximumThreads,
                maximumQueueSize = properties.maximumQueueSize,
                name = properties.threadPoolName
            )
        }

        val optionalCallable = OptionalCallable(context.job)
        val configureRequestAttributes = SpringServletRequestAttributesCallable(optionalCallable)
        val configureLocale = SpringLocaleContextCallable(configureRequestAttributes)
        val securityCallable = DelegatingSecurityContextCallable(configureLocale, context.securityContext)

        return taskCoordinator.execute(securityCallable)
            .timeout(Duration.ofMillis(context.configuration.timeoutInMillis.toLong()))
    }
}

class OptionalCallable(private val inner: RequestHandlerJob) : Callable<Optional<Any>> {
    override fun call(): Optional<Any> = Optional.ofNullable(inner.call())
}

The ThreadPoolPropertiesCalculator calculates how concurrent threads and how big the requests queue should be for particular tenant or tenants group. Then for each tenant group, in particular a single tenant, we create a TenantTaskCoordinator responsible for calculating and enforcing limits on concurrently handled requests. Further down we decorate the Callable representing the actual request handling with security delegation, locale configuration and request attributes setup. Finally, we ask the TenantTaskCoordinator to execute the decorated job with a configured timeout.

The last piece of the puzzle, namely TenantTaskCoordinator requires a separate blog post so stay tuned.

Request timeouts in Spring MVC

Tue, 28 Nov 2017 22:14:00 GMT

As we saw previously, we only have limited options to configure maximum time a request processing can take in Spring MVC. In this post I will show how to enforce such timeout through a custom Servlet Filter.

Request timeout Servlet Filter

Without further ado let us dive right into a sample filter implementation in Kotlin:

@Component
@Order(Ordered.HIGHEST_PRECEDENCE)
class TimeoutFilter : OncePerRequestFilter() {
    override fun doFilterInternal(request: HttpServletRequest, response: HttpServletResponse, filterChain: FilterChain) {
        val completed = AtomicBoolean(false)
        val requestHandlingThread = Thread.currentThread()
        val timeout = timeoutsPool.schedule({
            if (completed.compareAndSet(false, true)) {
                requestHandlingThread.interrupt()
            }
        }, 5, TimeUnit.SECONDS)

        try {
            filterChain.doFilter(request, response)
            timeout.cancel(false)
        } finally {
            completed.set(true)
        }
    }

    companion object {
        private val timeoutsPool = Executors.newScheduledThreadPool(10)
    }

    @ResponseStatus(HttpStatus.INTERNAL_SERVER_ERROR)
    class TimeoutException(message: String) : java.util.concurrent.TimeoutException(message)
}

The above code declares a Servlet Filter that will interrupt thread processing a request after 5 seconds. There are couple of interesting points about how it works.

@Order(Ordered.HIGHEST_PRECEDENCE) puts the Filter at the beginning of filter chain
val completed = AtomicBoolean(false) denotes whether the request processing completed.
val timeoutsPool = Executors.newScheduledThreadPool(10) creates a thread pool responsible for running timeouts. The newScheduledThreadPool creates a thread pool that is efficient at running delayed tasks.
timeoutsPool.schedule({ ... }) schedules a code that will interrupt requestHandlingThread after 5 seconds
completed.compareAndSet(false, true) updates the completed flag in a thread safe fashion

Testing request timeout Servlet Filter

For the test purposes let us create a simple Spring Boot MVC application written in Kotlin:

@SpringBootApplication
@EnableWebMvc
class Application {
    companion object {
        @JvmStatic
        fun main(args: Array<String>) {
            SpringApplication.run(Application::class.java)
        }
    }
}

@RestController
class TimeoutController {
    @GetMapping("/timeout")
    fun timeout(@RequestParam(required = false) timeoutInMillis: Long?): ResponseEntity<*> {
        Thread.sleep(timeoutInMillis ?: 1000)
        return ResponseEntity.ok("completed")
    }
}

The TimeoutController will sleep for an amount of time given in a parameter. Let’s simulate a short request with httpie:

http :8080/timeout timeoutInMillis==2000

HTTP/1.1 200 
Content-Length: 9
Content-Type: text/plain;charset=ISO-8859-1
Date: Mon, 27 Nov 2017 12:19:03 GMT

completed

This was the happy path. Now let’s try a timeout path:

http  :8080/timeout timeoutInMillis==6000       

HTTP/1.1 500 
Connection: close
Content-Type: application/json;charset=UTF-8
Date: Mon, 27 Nov 2017 12:21:30 GMT
Transfer-Encoding: chunked

{
    "error": "Internal Server Error",
    "exception": "java.lang.InterruptedException",
    "message": "sleep interrupted",
    "path": "/timeout",
    "status": 500,
    "timestamp": 1511785290518
}

As you can see in the exception message, we see that the Thread.sleep in the controller action has been interrupted 🎉

A word of warning

The above Servlet Filter will not work if we use Async Servlet Filters. When using Async Servlet Filter there is typically more than 1 thread that handles a request hence the above approach will not work. Having said that if you use Async Servlet Filter there already is a way to apply a timeout that is defined by the API. Another important point is to check how the request processing thread pool handles interrupted threads. As we have discussed earlier, the concrete implementation of thread pool used to process request depends on servlet container and configured used in the application. We should make sure that the interrupted thread is eventually replaced with a new thread by the pool so that timeouts do not change the effective thread pool size.

Request timeouts in Spring MVC

Tue, 21 Nov 2017 22:14:00 GMT

Last time we reviewed how to configure HTTP client timeouts. This time let us focus on the other side of the HTTP request i.e. server. There is pretty much always a thread pool involved when we write a Spring MVC application. The thread pool configuration will vary depending on particular servlet container (Tomcat, Undertow, Jetty) so we have to watch out for subtle differences. However, most if not all of them will use a thread pool with fixed maximum size. As we already know, a pool of resources might get exhausted. In particular, a thread pool is more likely to get exhausted if we do not control timeouts diligently.

Threads involved in a Spring MVC request handling

A typical servlet container will use one or more thread pools to handle a request. In particular one of the thread pools is used to execute the Spring MVC part of request handling. Let us call this thread pool the request worker thread pool. The request worker thread pool will have a default maximum size:

Tomcat: server.tomcat.max-threads controlling maxThreads with a default of 200
Undertow: server.undertow.worker-threads controlling WORKER_TASK_CORE_THREADS with a default of availableProcessors() * 8
Jetty: There is no Spring configuration property available currently. One can customize the Jetty Thread Pool through code and jetty specific configuration though. The default maximum number of worker threads is 200.

What happens when the request processing thread pool is empty?

Once the request processing thread pool is empty, the servlet container will typically queue the requests. The queue is processed by the request processing thread pool. Queueing up requests is consuming server memory and sockets thus there typically is a limit after which a new incoming request is going to be immediately rejected.

Tomcat: server.tomcat.accept-count Maximum queue length for incoming connection requests when all possible request processing threads are in use. The default value is 100.
Undertow: As far as I can tell by default the requests will be queued and the only bound is system capacity. There is Request Limiting Handler available though that allows configuring maximum concurrent requests as well as maximum queue size.
Jetty: By default will queue requests using unbounded queue. You can configure it though as documented:

<Configure id="Server" class="org.eclipse.jetty.server.Server">
    <Set name="ThreadPool">
      <New class="org.eclipse.jetty.util.thread.QueuedThreadPool">
        <!-- specify a bounded queue -->
        <Arg>
           <New class="java.util.concurrent.ArrayBlockingQueue">
              <Arg type="int">6000</Arg>
           </New>
      </Arg>
        <Set name="minThreads">10</Set>
        <Set name="maxThreads">200</Set>
        <Set name="detailedDump">false</Set>
      </New>
    </Set>
</Configure>

Queuing requests is necessary in the most commons scenarios to handle temporary spikes in load. For example, if your application can handle 100 requests per second, and if you can allow it to recover from one minute of excessive high load, you can set the queue capacity to 60*100=6000.

Let us assume that the thread pool (max) size is 100 and that on average a request takes 1 second to process. In such server we can thus handle 100 requests per second (rps). Any requests over the rps limit is going to be queued. Now imagine we have a single type of request that for some reason takes much longer to process than usual e.g. 120 seconds due to some dependent service issue. When such request is processed, it will first block subsequent requesting processing threads until all of them are busy waiting. Depending on the limit of queue size and system capacity our server will soon start rejecting all new requests. It is worth noting that the slow requests are also going to be put in queue after thread pool capacity is reached.

One of the ways to mitigate the issue and speed up system recovery is to apply timeouts. When a timeout for a particular request elapses ideally few things should happen:

the client should be notified about the error (503, 504 or 408 depending on the use case)
the request should be removed from the processing queue
the thread processing the requests should be interrupted

Let’s review what options are available.

Tomcat has Stuck Thread Detection Valve:

This valve allows to detect requests that take a long time to process, which might indicate that the thread that is processing it is stuck. Additionally it can optionally interrupt such threads to try and unblock them.

The valve has 2 configuration options:

threshold: Minimum duration in seconds after which a thread is considered stuck. Default is 600 seconds.
interruptThreadThreshold: Minimum duration in seconds after which a stuck thread should be interrupted to attempt to “free” it.

AFAIK the valve only applies to requests that did start processing by the thread pool.

Undertow and Jetty do not allow for setting a request timeout directly. They both do have idle connection detection and can timeout it accordingly. Unfortunately since HTTP/2 multiplexing the timeouts options may not be suitable to timeout a single request.
In Spring MVC there is no way to configure a timeout unless you use async method. With async method one can use spring.mvc.async.request-timeout= to set amount of time (in milliseconds) before asynchronous request handling times out. However, using Async Servlet with Spring MVC requires changing the controller methods return types.

There is no standard request timeout configuration

There are only couple of options available to set encompass request handling with a timeout. This is partially due to historical reasons. The servlet container specification did not consider timeouts until Async Servlet was defined. Another reason is that the there is no way to safely stop a thread that a framework could use. The application code needs to cooperate to safely terminate the request handling execution. In the next post we will show how to add a request timeout to a Spring MVC application.

HTTP client timeouts

Sun, 05 Nov 2017 22:14:00 GMT

We have already touched upon the importance of timeouts and described most important related JDBC knobs. The next aspect of timeouts I would like to focus on is using API clients. Specifically HTTP clients which are by far the most popular. We will review couple of popular HTTP client libraries and their configuration regarding timeouts.

HttpURLConnection timeouts

HttpURLConnection available since JDK 1.1 has gained the ability to timeout its network communication in version JDK 5. The 2 available timeouts setConnectionTimeout, setReadTimeout control how long to wait until connection is established and how long to wait for a data from the server respectively. The default values are infinite ‼️.

Apache HttpClient timeouts

HttpClient from Apache HttpComponents suite has been a standard choice for http communication. It is a mature project, with rich API that fills many HttpURLConnection shortcomings e.g. connection pooling. Many of the APIs have been deprecated e.g. DefaultHttpClient, org.apache.http.params.CoreConnectionPNames hence one needs to be careful when setting the timeouts they fallback to system defined socket level defaults.

There are 3 timeouts settings available:

val requestConfig = RequestConfig.custom()
    // Determines the timeout in milliseconds until a connection is established.
    .setConnectTimeout(5_000) 
    // Defines the socket timeout in milliseconds,
    // which is the timeout for waiting for data or, put differently,
    // a maximum period inactivity between two consecutive data packets).
    .setSocketTimeout(5_000)
    // Returns the timeout in milliseconds used when requesting a connection
    // from the connection manager.
    .setConnectionRequestTimeout(2_000)
    .build()

The requestConfig can be further used as a default for an HttpClient instance:

val httpClient = HttpClients.custom()
    .setDefaultRequestConfig(requestConfig)
    .build()

It is also possible to configure each request separately:

val get = HttpGet("http://httpbin.org/get").apply { 
    config = requestConfig
}
httpClient.execute(get)

OkHttp

OkHttp is my favorite HTTP & HTTP/2 client for Android and Java applications. It is efficient and has good configuration defaults. There are 3 timeout settings available:

val client = OkHttpClient.Builder()
    // Sets the default connect timeout for new connections.
    .connectTimeout(5, TimeUnit.SECONDS)
    // Sets the default read timeout for new connections.
    .readTimeout(10, TimeUnit.SECONDS)
    // Sets the default write timeout for new connections.
    .writeTimeout(20, TimeUnit.SECONDS)
    .build()

All connectTimeout, readTimeout and writeTimeout default to 10 seconds 👍.

XMLHttpRequest and Fetch API timeouts

XMLHttpRequest is the standard foundation of network communication of Web application for over 10 years now. Nowadays it is being replaced with Fetch API but it still is, and will continue to be, the most popular for couple of years. There is only a single timeout configuration available in XMLHttpRequest:

The XMLHttpRequest.timeout property is an unsigned long representing the number of milliseconds a request can take before automatically being terminated. The default value is 0, which means there is no timeout.

Default is infinite ‼️

Since the default value is not configured we should diligently set the timeout in our code! It may be tempting to think that client side timeout is not so important compared to the one on the server. This is a questionable attitude to say the least. We need to keep in mind that there is a hard limit on the number of connections a browser will make to a single domain which is very important if we use HTTP 1.*. When we reach maximum number of concurrently opened connections, any new XMLHttpRequest is going to be queued indefinitely. The limit value varies in browsers and the recent RCF relaxes it. HTTP/2 alleviates the issue with connection multiplexing nonetheless its adoption is still low. According to w3techs it is about 20% as of today. The timeout value used in XMLHttpRequest is even more important in Single Page Applications. In SPAs the XMLHttpRequest without a timeout can live for as long as server and intermediate network parties allow effectively blocking all subsequent network calls.

Fetch API is meant to replace XMLHttpRequest. It is thus sad that the ability to timeout a request has not yet made it into the standard. Currently there is no standard way to enforce a timeout. There are couple of GitHub issues active: Add timeout option, Add option to reject the fetch promise automatically after a certain time elapsed that go over potential solutions. There was a proposal for cancelable promises which had been withdrawn after lots of discussion and lack of consensus. A brand new way has recently been implemented by Edge and Firefox allows one to timeout a Fetch API call 🎉 through the DOM standardized AbortController. Hopefully it will get into the Fetch API standard soon.

const controller = new AbortController();
const signal = controller.signal;

setTimeout(() => controller.abort(), 5000);

fetch(url, { signal }).then(response => {
  return response.text();
}).then(text => {
  console.log(text);
});

URLSession timeouts

URLSession is the successor to NSURLConnection that underlays most if not all iOS http clients e.g. Alamofire. There are 2 main timeout values to configure both of which have default values available via URLSessionConfiguration.default:

let sessionConfig = URLSessionConfiguration.default
sessionConfig.timeoutIntervalForRequest = 20.0
sessionConfig.timeoutIntervalForResource = 40.0

let session = URLSession(configuration: sessionConfig)

Fortunately there are default values configured:

timeoutIntervalForRequest:

This property determines the request timeout interval for all tasks within sessions based on this configuration. The request timeout interval controls how long (in seconds) a task should wait for additional data to arrive before giving up. The timer associated with this value is reset whenever new data arrives.

The default value is 60.
timeoutIntervalForResource:

This property determines the resource timeout interval for all tasks within sessions based on this configuration. The resource timeout interval controls how long (in seconds) to wait for an entire resource to transfer before giving up.

The default value is 7 days.

Note that timeoutIntervalForResource is a higher level timeout that what we have considered in other HTTP clients. It encompasses retries and or request timeouts hence has a large default.

Summary

Many of HTTP clients do not have a good default timeout configuration. Hence, if you care about your application resource usage and system stability you have to carefully review and configure timeouts where applicable. It is reassuring to see that modern HTTP clients e.g. OkHttp and URLSession have a short but sane default.

Database timeouts

Tue, 31 Oct 2017 22:14:00 GMT

Last time I have outlined the importance of timeouts. Without a carefully considered timeouts our application can become unresponsive easily. In this post I will focus on configuring various timeouts related to interaction with database. I am going to focus specifically on relational databases. The principles and practices however can be applied equally well to other types of databases.

Different kinds of timeouts

Asking a database for results of a query is one of the most common activities a back end application will do. Let us decompose this simple task into steps:

Establish database connection(s) in pool
Take the connection out of the pool
Validate the acquired connection
Send statement(s) to database
Read query results

Each of the above steps can involve a specific timeout configuration. The details depend on particular technology stack and type of database we are querying.

Establish database connection(s) in pool

Dealing with raw database connections is almost always done with the help of connection pool. Establishing a connection to database is very expensive compared to running a simple statement. The pool alleviates this cost by reusing connections for as long as needed.

The first timeout is the maximum duration until a database connection is established. In JDBC this can be controlled by:

connectTimeout in MySQL JDBC driver

Timeout for socket connect (in milliseconds), with 0 being no timeout. Only works on JDK-1.4 or newer. Defaults to ‘0’.

The default is infinite ‼️
socketTimeout in MySQL JDBC driver

Timeout (in milliseconds) on network socket operations (0, the default means no timeout).
loginTimeout in MySQL JDBC driver

Sets the maximum time in seconds that a driver will wait while attempting to connect to a database once the driver has been identified.

The default value 0 means infinite ‼️
loginTimeout in PostgreSQL JDBC driver:

Specify how long to wait for establishment of a database connection. The timeout is specified in seconds.

Default is infinite ‼️
connectTimeout in PostgreSQL JDBC driver:

The timeout value used for socket connect operations. If connecting to the server takes longer than this value, the connection is broken. The timeout is specified in seconds and a value of zero means that it is disabled.
socketTimeout in PostgreSQL JDBC driver:

The timeout value used for socket read operations. If reading from the server takes longer than this value, the connection is closed. This can be used as both a brute force global query timeout and a method of detecting network problems. The timeout is specified in seconds and a value of zero means that it is disabled.

You probably have noticed a recurring theme above: default timeouts are either infinite or disabled at the driver level. In case of socketTimeout and connectTimeout there can still be a system level timeout involved both on Linux and Windows. However, those only work on blocking send and receive operations and how the JDBC driver interacts with the socket is an implementation detail for the most part.

In order to demonstrate how the above timeouts work in practice we will use the following test cases:

class JdbcTimeoutTest {
    @Test
    fun `mysql getConnection`() {
        val mysqlDataSource = mysqlDataSource()

        assertTimeoutPreemptively(Duration.ofMinutes(3)) {
            useResource { mysqlDataSource.connection }
        }
    }

    @Test
    fun `postgresql getConnection`() {
        val mysqlDataSource = postgreSQLDataSource()

        assertTimeoutPreemptively(Duration.ofMinutes(3)) {
            useResource { mysqlDataSource.connection }
        }
    }

    fun useResource(resourceProvider: () -> AutoCloseable) {
        val start = Instant.now()
        try {
            resourceProvider().use {
                println("Completed in ${Duration.between(start, Instant.now())}")
            }
        } catch (e: Exception) {
            println("Error $e after ${Duration.between(start, Instant.now())}")
        }
    }

    fun mysqlDataSource(): MysqlDataSource {
        return MysqlDataSource().apply {
            this.setURL("jdbc:mysql://localhost:3306/database")
            this.user = "user"
            this.setPassword("password")
        }
    }

    fun postgreSQLDataSource(): PGSimpleDataSource {
        return PGSimpleDataSource().apply {
            this.user = "user"
            this.password = "password"
            this.databaseName = "database"
            this.serverName = "localhost"
        }
    }
}

To simulate a misbehaving database server we’ll use netcat listening on standard MySQL and PostgreSQL port e.g.:

nc -k -l 3306 # listen on MySQL port, PostgreSQL uses 5432 by default

Both of the above tests will fail due to assertTimeoutPreemptively.

The most appropriate candidate to use for establishing connection is loginTimeout. This works in PostgreSQL but does not with MySQL. Apparently the MySQL JDBC driver in versions 5.1, 6.0 and 8.0 implement the method as Noop. Interestingly it is possible to force MySQL driver to respect the timeout when it is set globally through a static method java.sql.DriverManager.setLoginTimeout.

A slightly less correct option is to use connectTimeout or socketTimeout. The socket level options work oblivious to database protocol hence it is impossible to set the timeout accurately for the whole establish connection operation. Additionally, the socketTimeout is applied to socket read operations not only to initial connection handshake. The fake netcat server is not suitable for testing the connectTimeout however we can use it for socketTimeout. The PostgreSQL driver correctly reported error after about 2 seconds after setting pgSimpleDataSource.socketTimeout = 2. Unfortunately settings socketTimeout in MySQL driver had no effect on the getConnection behavior. Interestingly no matter what value I’ve set the socketTimeout to, the error was thrown after about 26 seconds. I have no idea why it behaves like that 🤔.

Be aware of the shortcomings of MySQL JDBC Driver.

Take the connection out of the pool

Reusing database connections gives the application great performance boost. However, writing an efficient and bug free database connection pool is no easy task thus we should all rely on proven solutions. In JVM world there are multiple choices when it comes to JDBC:

Hikari Claims to be the fastest and has limited number of configuration knobs and sane defaults. My favorite by far.
DBCP 2 A recently resurrected project which has a potential of being applicable to all resources pools with its commons-pool2 module.
Tomcat JDBC Connection Pool Commonly used with lots of configuration options. Came to be as a replacement of dbcp.

When there are no connections available in the pool, the code asking for it needs to wait until one is available. The amount of time a thread is blocked waiting for a connection needs to be considered carefully. It is important to note that there are 2 situations we need to consider. The first one is when the pool has reached its maximum size and all connections are being used already. This is when there is very little actual work required to acquire a connection. The second case is when all currently opened connections are in use but the pool is allowed to create a new connection because it is not yet full. Here we need to keep in mind that time to establish a connection to the database may easily be around 200ms hence timeout should not be too short. Below you’ll find how to configure the timeout in the mentioned connection pools:

Hikari: connectionTimeout

This property controls the maximum number of milliseconds that a client (that’s you) will wait for a connection from the pool. If this time is exceeded without a connection becoming available, a SQLException will be thrown. Lowest acceptable connection timeout is 250 ms. Default: 30000 (30 seconds)
DBCP: maxWaitMillis

The maximum number of milliseconds that the pool will wait (when there are no available connections) for a connection to be returned before throwing an exception, or -1 to wait indefinitely.

Default is infinite ‼️
Tomcat: maxWait

(int) The maximum number of milliseconds that the pool will wait (when there are no available connections) for a connection to be returned before throwing an exception. Default value is 30000 (30 seconds)

My rule of thumb is to set this timeout be under 5 seconds.

Validate the acquired connection

A database connection can be opened for several hours or even days. However, because there is network involved there are numerous cases where a socket that is seemingly open on the client side may in fact be part of a broken connection. A well behaving connection pool should avoid handing such connection to an application code. A common strategy to alleviate the problem is to test the connection just before it is taken out of the pool. In the past the test was performed using a simple SQL query e.g. SELECT 1. Nowadays there is a isValid method available on the JDBC Connection itself which moves the responsibility to the driver itself:

Returns true if the connection has not been closed and is still valid. The driver shall submit a query on the connection or use some other mechanism that positively verifies the connection is still valid when this method is called.

Hikari: validationTimeout:

This property controls the maximum amount of time that a connection will be tested for aliveness. This value must be less than the connectionTimeout. Lowest acceptable validation timeout is 250 ms. Default: 5000
DBCP: validationQueryTimeout:

The timeout in seconds before connection validation queries fail. If set to a positive value, this value is passed to the driver via the setQueryTimeout method of the Statement used to execute the validation query.

Default is infinite ‼️
Tomcat: validationQueryTimeout:

(int) The timeout in seconds before a connection validation queries fail. This works by calling java.sql.Statement.setQueryTimeout(seconds) on the statement that executes the validationQuery. The pool itself doesn’t timeout the query, it is still up to the JDBC driver to enforce query timeouts. A value less than or equal to zero will disable this feature. The default value is -1.

Default is infinite ‼️

Send statement(s) to database and read query results

We have finally arrived at the most common usage. Every query that we send to a database should have a timeout configured either at the statement level or at the transaction level. When it comes to individual statements, there is setQueryTimeout available:

Sets the number of seconds the driver will wait for a Statement object to execute to the given number of seconds. By default there is no limit on the amount of time allowed for a running statement to complete. If the limit is exceeded, an SQLTimeoutException is thrown. A JDBC driver must apply this limit to the execute, executeQuery and executeUpdate methods.

Additionally, it’s up to the driver to decide what the above timeout means exactly:

Note: JDBC driver implementations may also apply this limit to ResultSet methods (consult your driver vendor documentation for details).

Note: In the case of Statement batching, it is implementation defined as to whether the time-out is applied to individual SQL commands added via the addBatch method or to the entire batch of SQL commands invoked by the executeBatch method (consult your driver vendor documentation for details).

A time required for a query to complete is very use case dependent thus we should not expect a sane default to be there. Instead, we need to ask ourself how long we are willing to wait for a query to complete. It is very easy to forget about this rule hence it is very handy to be able to set this timeout globally:

DBCP: defaultQueryTimeout

defaultQueryTimeout null If non-null, the value of this Integer property determines the query timeout that will be used for Statements created from connections managed by the pool. null means that the driver default will be used.
Tomcat: queryTimeout available through QueryTimeoutInterceptor

(int as String) The number of seconds to set for the query timeout. A value less than or equal to zero will disable this feature. The default value is 1 seconds.
Hikari: Not available but fairly easy to add by wrapping a DataSource e.g.:

class CustomTimeoutsDataSource(val innerDataSource: DataSource, private val queryTimeoutProperties: DataSourceConfiguration.QueryTimeoutProperties) : DataSource by innerDataSource {

    override fun getConnection(username: String?, password: String?) = configureTimeouts(innerDataSource.getConnection(username, password))
    override fun getConnection() = configureTimeouts(innerDataSource.connection)

    private fun configureTimeouts(connection: Connection):Connection = CustomTimeoutsConnection(connection, queryTimeoutProperties)

    private class CustomTimeoutsConnection(val innerConnection: Connection, private val queryTimeoutProperties: DataSourceConfiguration.QueryTimeoutProperties)
        : Connection by innerConnection {

        private fun <T : Statement> configure(prepareStatement: T): T {
            //0 means no timeout
            val desiredTimeout = queryTimeoutProperties.statementQueryTimeoutInSeconds ?: 0
            prepareStatement.queryTimeout = desiredTimeout
            LOG.trace("Configure timeout {} seconds for statement {}", desiredTimeout, prepareStatement)
            return prepareStatement
        }

        override fun prepareStatement(sql: String?, autoGeneratedKeys: Int) = configure(innerConnection.prepareStatement(sql, autoGeneratedKeys))
        override fun prepareStatement(sql: String?, resultSetType: Int, resultSetConcurrency: Int, resultSetHoldability: Int) = configure(innerConnection.prepareStatement(sql, resultSetType, resultSetConcurrency, resultSetHoldability))
        override fun prepareStatement(sql: String?) = configure(innerConnection.prepareStatement(sql))
        override fun prepareStatement(sql: String?, columnNames: Array<out String>?) = configure(innerConnection.prepareStatement(sql, columnNames))
        override fun prepareStatement(sql: String?, resultSetType: Int, resultSetConcurrency: Int) = configure(innerConnection.prepareStatement(sql, resultSetType, resultSetConcurrency))
        override fun prepareStatement(sql: String?, columnIndexes: IntArray?)= configure(innerConnection.prepareStatement(sql, columnIndexes))
        override fun prepareCall(sql: String?) = configure(innerConnection.prepareCall(sql))
        override fun prepareCall(sql: String?, resultSetType: Int, resultSetConcurrency: Int) = configure(innerConnection.prepareCall(sql, resultSetType, resultSetConcurrency))
        override fun prepareCall(sql: String?, resultSetType: Int, resultSetConcurrency: Int, resultSetHoldability: Int) = configure(innerConnection.prepareCall(sql, resultSetType, resultSetConcurrency, resultSetHoldability))
        override fun createStatement() = configure(innerConnection.createStatement())
        override fun createStatement(resultSetType: Int, resultSetConcurrency: Int) = configure(innerConnection.createStatement(resultSetType, resultSetConcurrency))
        override fun createStatement(resultSetType: Int, resultSetConcurrency: Int, resultSetHoldability: Int) = configure(innerConnection.createStatement(resultSetType, resultSetConcurrency, resultSetHoldability))

        override fun toString(): String {
            return "CustomTimeoutsConnection(innerConnection=$innerConnection)"
        }
    }

    companion object {
        private val LOG = LoggerFactory.getLogger(CustomTimeoutsDataSource::class.java)
    }
}

The JDBC level queryTimeout is enforced at the application code side i.e. there’s a code executed after the timeout elapses which stops the query execution. Recent releases of both MySQL and PostgreSQL offer a database server level statement timeout capabilities.

MySQL: MAX_EXECUTION_TIME

The MAX_EXECUTION_TIME hint is permitted only for SELECT statements. It places a limit N (a timeout value in milliseconds) on how long a statement is permitted to execute before the server terminates it:
PostgreSQL: statement_timeout

Abort any statement that takes more than the specified number of milliseconds, starting from the time the command arrives at the server from the client. If log_min_error_statement is set to ERROR or lower, the statement that timed out will also be logged. A value of zero (the default) turns this off.

Setting statement_timeout in postgresql.conf is not recommended because it would affect all sessions.

If you are using a JPA provider like Hibernate, you might be urged to use javax.persistence.query.timeout. However, from my experience using Hibernate this timeout, when configured globally, is enforced in some uses cases and completely ignored in others. There were multiple bugs reported related to this feature: Bug 303360, HHH-9929, “Query timeout in persistence.xml doesn’t work” and some of them are still not addressed.

There is no transaction scoped timeout available in JDBC. However, one can still apply the timeout in the application code through e.g. @Transactional.timeout.

Keep all timeouts short

The best timeout is a short one. It is often tempting to increase a query or wait timeout in face of performance or throughput problems. However, doing so will increase the amount of resources blocked on the server and is thus a rarely a good choice. Blocking more and more resources on server e.g. threads, may at some point, cause the entire server to collapse abruptly. I keep the timeouts as short as possible especially when certain API is called often. If you are looking for a good read on the topic I highly recommend Release It! by Michael T. Nygard. This books covers many resiliency related topics including timeouts and provides strategies to avoid increasing them.

The importance of timeouts

Mon, 23 Oct 2017 22:14:00 GMT

Timeouts are not an exciting thing to talk about. They do not add immediately perceivable value. They are difficult to ~~guess~~ get right and force one to consider problems that are hard to solve. In fact, in my experience, the timeout is only ever considered when our software stops working or is about to. That is an enormous shame since, in my opinion, carefully applied timeouts can vastly improve software resiliency.

An example application

Let us consider a simplistic example of a Spring Boot application generated using Spring Initializr. The application will only expose actuator API which by default define a health check endpoint. Our example will also have a mail module configured.

The dependencies section of build.gradle:

dependencies {
	compile('org.springframework.boot:spring-boot-starter-actuator')
	compile('org.springframework.boot:spring-boot-starter-mail')
	compile('org.springframework.boot:spring-boot-starter-web')
	compile("org.jetbrains.kotlin:kotlin-stdlib-jre8:${kotlinVersion}")
	compile("org.jetbrains.kotlin:kotlin-reflect:${kotlinVersion}")
}

A typical health check endpoint verifies that application integration points work correctly. If the service talks to database then a connection is established and verified. A free disk space is checked. If the service sends emails through SMTP then a connection to a server is established.

The health checks are auto discovered and enabled when you include Spring Boot Actuator. By default, /health path is used for the endpoint. The SMTP server host, port and credentials obviously need to be configured. At minimum host entry is required e.g. spring.mail.host=localhost. For the debugging purpose one can disable actuator security with management.security.enabled=false to verify what health checks are performed by actuator.

A last part of our example is the trivial application code:

@SpringBootApplication
class Application {
    companion object {
        @JvmStatic
        fun main(args: Array<String>) {
            SpringApplication.run(Application::class.java, *args)
        }
    }
}

When you request /health the API will return response similar to:

HTTP/1.1 200 
Content-Type: application/vnd.spring-boot.actuator.v1+json;charset=UTF-8
Date: Mon, 23 Oct 2017 08:08:32 GMT
Transfer-Encoding: chunked
X-Application-Context: application

{
    "diskSpace": {
        "free": 105755779072,
        "status": "UP",
        "threshold": 10485760,
        "total": 499046809600
    },
    "mail": {
        "location": "localhost:-1",
        "status": "UP"
    },
    "status": "UP"
}

An application health API, like the one in our example, is often hooked into external monitoring software. The monitor asks the target application about its health in regular intervals e.g. every 5 seconds.

Shooting yourself in the foot

The above example has an issue that can kill production server. More importantly, other metrics that are usually monitored e.g. CPU and memory usage will not warn about upcoming, dreadful service stall. The application will also not suffer from an enormous number of requests or emails being sent.

Imagine that the health endpoint is checked every 5 seconds and that there is intermittent issue with the SMPT server. The health endpoint will rightfully try to connect to the SMTP server and from time to time respond with error. From my experience, when a health check is introduced it typically takes a while to tweak the monitor thresholds so that we get rid of false alarms. It is thus very easy to ignore intermittent errors when we think they are caused by too sensitive thresholds. However, the ignored errors can after a while cause our server to stop responding to any request.

Why this can happen you ask and I answer. There is no timeout configured!

The mail server health check uses javax.mail.Service.connect under the hood. For a variety of reasons an attempt to establish a TCP connection can take arbitrary longer than usual. Unfortunately the default timeouts used by the javax.mail.* are infinite. A thread that waits for the connection to be established cannot serve other requests even though it barely uses any CPU. The default maximum thread pool size used by embedded Tomcat in Spring Boot application is 200. Assuming that the blocked connection attempt happens twice an hour our application will stop working after 4 days.

Never use infinite timeouts

As you can see it is very easy to miss a need for a timeout to be configured. To be fair the Spring Boot documentation states clearly:

In particular, certain default timeout values are infinite and you may want to change that to avoid having a thread blocked by an unresponsive mail server:
spring.mail.properties.mail.smtp.connectiontimeout=5000
spring.mail.properties.mail.smtp.timeout=3000
spring.mail.properties.mail.smtp.writetimeout=5000

In my option any library or framework should either force the programmer to configure the timeout or have some sensible default. Unfortunately this is not always possible to introduce them later on without breaking changes hence we should check what are the timeouts used when calling any external service.

Timeouts needed everywhere

Imagine a controller action method that inserts a single row into a database. Let us further assume that the endpoint is called 50 times per second and it typically takes 100ms to complete. Things work well until we encounter an intermittent sloppiness of the database and now the insert takes 2 seconds to complete. The clients calling the API do not slow down. More request threads are blocked and more database connections are taken out of the pool. Soon as all database connections are in use and all other API endpoints start to fail. This is an example of cascading failure i.e. a problem in one component propagating to others. It is easier to avoid such issues when there is a timeout configured on the controller action and the interaction with the database.

Every API endpoint should have a timeout configured if we want our services to be resilient. Unfortunately this rule is easy to neglect. Moreover, some frameworks do not even expose such ability. Even in the immensely popular Spring Mvc I could not find a way to set such timeout other than using an async method. Fortunately there are libraries e.g. Hystrix that tackle that problem and can be integrated easily.

Just to recap here is a short and incomplete list of cases where a timeout should be configured:

a controller action
a database query, statement
a database pool interaction
a thread pool interaction
an API client e.g. HTTP, SOAP, SMTP

I will describe how to deal with the cases in the following posts.

How to clear database in Spring Boot tests?

Fri, 13 Oct 2017 22:14:00 GMT

Nowadays using a production like database in unit¹ tests is a common practice. Calling a real database can increase our confidence that a tested code actually works. Having said that a database, by its very nature, brings external state into a test that will affect its behavior, hence we need to pay special attention to prepare the test execution. There are couple of ways to handle the database state in tests and I’m going to describe an approach I like most.

{: .center-image}

Problems with Spring Boot Transactional tests

Spring Boot offers many helpers to make testing application easier. Among many you can use a @DataJpaTest which by default will configure an in-memory embedded database. You can use a production type database in tests by adding @AutoConfigureTestDatabase(replace=Replace.NONE) like so:

@RunWith(SpringRunner.class)
@DataJpaTest
@AutoConfigureTestDatabase(replace=Replace.NONE)
public class ExampleRepositoryTests {

    // ...

}

The @DataJpaTest uses @Transactional under the hood. A test is wrapped inside a transaction that is rolled back at the end. This means that when using e.g. Hibernate one needs to pay special attention to how the tested code is written. As shown in the Java example below, a manual flush is indeed required:

@RunWith(SpringRunner.class)
@ContextConfiguration(classes = TestConfig.class)
@Transactional
public class HibernateUserRepositoryTests {
    ...
    @Test
    public void createUser() {
        // track initial state in test database:
        final int count = countRowsInTable("user");

        User user = new User(...);
        repository.save(user);

        // Manual flush is required to avoid false positive in test
        sessionFactory.getCurrentSession().flush();
        assertNumUsers(count + 1);
    }
}

Using @Transactional annotation on tests is certainly easy but I still don’t use it for the following reasons:

The production code is using transactions with different scope.
It is easy to forget about a flush and thus have false positive in test.
On failure and when debugging it is hard to see what values were actually saved in db.
It is much harder to write tests of production code that requires a transaction to be committed.
The test code needs to be more tightly coupled to production code and we all know that it hinders refactoring.

Cleaning database with SQL

In tests involving a database I reset its state before each test using plain old SQL. This makes the test code less dependent on how a transaction is scoped inside production code. Furthermore, one can easily review the values saved after a test failure. It turns out it is easy to write a JUnit @Rule or BeforeEachCallback that will remove all rows from all tables. Moreover, we can do so without hard coding table names which would increase maintenance cost.

Let’s start with defining a @Rule in Kotlin in that will be called before each test:

import org.junit.rules.ExternalResource
import org.springframework.stereotype.Component
import javax.sql.DataSource

@Component
class DatabaseCleanerRule(private val dataSource: DataSource) : ExternalResource() {

    override fun before() {
        if (databaseCleaner == null) {
            // Consider inspecting dataSource to check if we are connecting to test database
            databaseCleaner = DatabaseCleaner(dataSource::getConnection)
        }
        databaseCleaner!!.reset()
    }

    companion object {
        internal var databaseCleaner: DatabaseCleaner? = null
    }
}

Consider inspecting dataSource to check if we are about to connect to test database and not one used for development. It is very easy to use incorrect Spring Profile and wipe out your development data. Ask me how I know?

We can use the DatabaseCleanerRule in a spring enabled test as any other JUnit rule e.g. @Rule @Inject lateinit var cleanerRule: DatabaseCleanerRule.

Notice that we’ve delegated the actual important work to DatabaseCleaner class defined in Kotlin below.

import com.practi.util.iterator
import org.slf4j.LoggerFactory
import java.sql.Connection
import java.sql.PreparedStatement
import java.sql.SQLException

class DatabaseCleaner(private val connectionProvider: () -> Connection) {
    private val tablesToExclude = mutableSetOf<String>()
    private var tablesForClearing: List<TableRef>? = null

    fun excludeTables(vararg tableNames: String) {
        tablesToExclude += tableNames.flatMap { listOf(it, it.toLowerCase()) }
    }

    fun reset() {
        if (notPrepared) {
            prepare()
        }
        executeReset()
    }

    private val notPrepared get() = tablesForClearing == null

    private fun prepare() {
        connectionProvider().use { connection ->
            val metaData = connection.metaData
            val tableRefs = metaData.getTables(connection.catalog, null, null, arrayOf("TABLE")).use { tables ->
                iterator(tables::next) { tables.getString("TABLE_NAME") }
                    .asSequence()
                    .filterNot(tablesToExclude::contains)
                    .map(::TableRef)
                    .toList()
            }

            tablesForClearing = tableRefs

            LOG.info("Prepared clean db command: {}", tablesForClearing)
        }
    }

    private fun executeReset() {
        try {
            connectionProvider().use { connection ->
                val reset = buildClearStatement(connection)
                val result = reset.executeBatch()
                result
            }
        } catch (e: SQLException) {
            val status = engineInnoDbStatus()
            LOG.error("Failed to remove rows because {}. InnoDb status: {}", e, status)
            throw e
        }
    }

    private fun engineInnoDbStatus(): String { ... }

    private fun buildClearStatement(connection: Connection): PreparedStatement {
        val reset = connection.prepareStatement("")
        reset.addBatch("SET FOREIGN_KEY_CHECKS = 0")
        tablesForClearing?.forEach { ref ->
            reset.addBatch("DELETE FROM `${ref.name}`")
        }
        reset.addBatch("SET FOREIGN_KEY_CHECKS = 1")
        return reset
    }

    data class TableRef(val name: String)

    companion object {
        private val LOG = LoggerFactory.getLogger(DatabaseCleaner::class.java)!!
    }
}

Notice that we’ve defined tablesToExclude set that allows us to omit certain tables. This comes in handy when you’re using a database migration tool that stores its state inside some table(s).

The JDBC metadata allows us to introspect schema regardless of the database vendor. The iterator is a tiny Kotlin function that aids consuming iterator like objects:

inline fun <T> iterator(crossinline next: () -> Boolean, crossinline value: () -> T): AbstractIterator<out T> = object : AbstractIterator<T>() {
    override fun computeNext() {
        if (next()) {
            setNext(value())
        } else {
            done()
        }
    }
}

The buildClearStatement constructs a large query that DELETEs all rows from each relevant table. The example above uses MySQL where it is very easy to disable foreign key checks. This is important since foreign keys would prevent rows to be removed unless we paid special attention to the order of removal. A more generic example of how to deal with referential integrity when clearing a database can be found in the Respawn project.

Last but not least, when a SQLException is thrown we log the exception accompanied with SHOW ENGINE INNODB STATUS. The status information can hint us about failure reason e.g. another test process executing against the same database or a rogue, runaway thread that locks some rows.

private fun engineInnoDbStatus(): String {
    return connectionProvider().use { connection ->
        connection.createStatement().executeQuery("SHOW ENGINE INNODB STATUS ").use {
            iterator(it::next) { it.getString("Status") }.asSequence().toList()
        }.joinToString(System.lineSeparator())
    }
}

The above examples show that it is not hard to manually reset the database. I’ve found that using this approach makes my tests more trustworthy and less coupled to the underlying persistence layer. In fact, we can easily switch e.g. from JPA to JdbcTemplate in a performance critical code area without a need to change a test.

¹ Whether it is actually unit or integration test is a different topic.

Simple Config Sections 1.2 released

Fri, 25 Dec 2015 10:21:00 GMT

About a week ago I’ve released another version of SimpleConfigSections. It’s marked 1.2.0 and you can install it through Install-Package SimpleConfigSections.

What is SimpleConfigSections?

SimpleConfigSections is a little library that simplifies defining .net configuration sections. As you can see in the mentioned msdn documentation the amount of boilerplate code is substantial. An equivalent example in SimpleConfigSections looks like follows:

As you can see the SimpleConfigSections version is more concise and a bit easier to write. More examples can be found on project’s github page.

Android Data Binding Part 2: Observer pattern to the bone

Mon, 27 Jul 2015 08:45:00 GMT

In the previous part I’ve described typical problems we have to face when developing applications on Android. I’ve also highlighted that some of them may be mitigated when data binding API is utilized properly. It’s time to dive into more details of how this promising API works.

Observer Pattern

At the root of many of solutions we find in today’s APIs is a design pattern. In case of the discussed Android API it is Observer Pattern applied to the bone. In fact this particular pattern is so common and powerful that some languages and runtimes (C# with .NET, Objective-C and Swift on iOS and Mac) provide neat support for it. You may wonder why it is important in the context of Android data binding? The answer is simple yet easy to ignore - memory management. The following diagram depicts dependencies of an observer pattern’s elements in the context of Android data binding: On the right side we have the observable part producing notifications about changes - an object implementing android.databinding.Observable interface. In the observer pattern events producer can have many events listeners - OnPropertyChangedCallback implementations. Android data binding provides implementations of events listeners taking care of property values changes as well as collection elements changes.

Note that the Observable object knows nothing about concrete OnPropertyChangedCallback implementations (Dependency Inversion Principle at its best). However, at runtime the Observable object will hold references to many OnPropertyChangedCallbacks instances. If you look further down the diagram you’ll see that in order for OnPropertyChangedCallback implementations provided by Android data binding library to update the state of the view component a reference to it is required. This means that although the view model knows nothing about the view components at compile time they will reference them at runtime.

How Android data binding library aids memory management?

As stated above in a simple implementation we would have a lightweight and testable view model retain expensive view widgets - thus making it heavy. In the Android context it means that even though an Activity got destroyed we could have other object still retaining its Views, trying to update them and preventing them from being garbage collected. Not so nice, huh?

If you look closer at how Android data binding is implemented you’ll immediately see that its OnPropertyChangedCallback holds only weak references to views. WeakPropertyListener, WeakListListener, WeakMapListener and ViewDataBinding.WeakListener make sure that the Observable object is not retaining views. This means that there’s no need for a developer to manually stop data binding in an activity onDestroy or a fragment onDestroyView methods. If you use RxJava you probably know that this extra step is tedious and requires a great deal of attention and boilerplate code. You can find out more about the problem in at least couple of issues on GitHub: 1, 2.

Because Android data binding library takes extra steps to make sure a view model will not cause views to be retained we can detach a view model’s lifecycle from that of an activity or a fragment. We now have an easy way to i.e. do a simple in-memory caching via singleton view models. More importantly, we can cross out at least one place when looking for a memory leak cause.

This article is cross-posted with my company blog

Android Data Binding Part 1: Why it is important

Mon, 20 Jul 2015 08:36:00 GMT

Android application code often suffers from being more verbose than it could be. As libraries such as Android Annotations and ButterKnife have shown that’s only partially due to tediousness of Java. The recently announced Android Data Binding library can remove at least part of the boilerplate code we need to write. Since I’ve always liked Presentation Model pattern (MVVM) this is very dear to my heart. However just getting rid of a tedious code is not the main reason I’m so happy to see the new API. Let’s recap on common issues developer faces on Android and then I’ll show how using mentioned patterns with new offering from Google can mitigate them.

Problems with classic approach

I’ll explain that step by step using a registration form screen example. We need couple of fields like first name, last name and an email address. To make the example a bit more interesting and to make UX better we’ll use floating label pattern. One of the simplest approaches without data binding (for first name only) might look like this:

Memory leaks

A seasoned Android developer will immediately spot that we’ve a potentially leaked activity here. The culprit is of course the anonymous inner class implementing RegisterApi callback (Action2), which in turn uses findViewById. However it’s not the mentioned method that retains the activity - in Java every non-static inner class will have implicit reference to the enclosing class instance. While this problem is so common I still haven’t found a concise solution in the Android SDK. There are however community driven libraries i.e. Otto and RxJava that can help tackle this problem.

To much code in Activity

Many of us, me including, are guilt of stuffing too much into Activity classes. Writing test and maintaining them can become a nightmare over time. The verbosity of Java only makes the matter worse. One way to slim activities down is to use Presenter pattern written either by hand or with the help of Mortar. Another solution is to encapsulate more logic into custom view classes used in layouts and using shared static helper methods with common code that calls Android API. Nevertheless, for a programmer that just starts with Android, it’s important that the framework provides guidance and samples that encourage separation of concerns - in my opinion up until now that was rarely the case.

Maintaining layout files

Many if not most screens in Android (and other platforms, too) are read-only. That is they are not forms and fields that user can change. Instead they offer neatly presented content with some way of interacting with it (as opposed to changing it directly). When using Android layout files we’re forced to use references (auto generated in R class by aapt) that allow setting view’s properties. This isn’t something necessarily bad especially if you need to heavily interact with a View. Having said that, how often you used findViewById (or @InjectView) just to set the content of TextView or had to refactor activity fields type because of a change in a layout definition?

Android Data Binding can help you.

The following code illustrates how registration form might look like with data binding:

The new RegisterActivity is similar to previous example. We inflate the layout differently to initialize data binding and get an instance of ActivityRegisterBinding class. The class, with the name derived from layout name, is generated by data binding and its main purpose is to provide a glue between a view and an observable view model. When using registerApi.register we only reference form variable thus avoid unnecessarily referencing the activity. The RegisterForm class is a view model for our layout and has all the data that activity_register layout requires.

The view model has 3 observable attributes marked with @Bindable annotation. We don’t strictly need a backing field as the getFirstNameLabelVisibility shows. What we do need is to fire change notification events, which is done via notifyPropertyChanged and notifyChange methods available on BaseObservable. The former method accepts a field id that is assigned by data binding processor in generated class BR (similarily to R class). Deriving from BaseObservable is not a must - to make a model bindable it has to implement android.databinding.Observable. Last but not least a layout file:

The data binding library, while still in beta, provides a nice toolset to remove the boilerplate code and improve Android code (unit) testability by extracting the logic from activity (or fragment). When used properly it can also save you from leaking resources. In the next post I’ll explain in more detail how it happens.

This article is cross-posted with my company blog

Handling different user types in android application

Sun, 12 Jul 2015 23:40:00 GMT

In the previous post you can read how to use Session object to maintain current user information through the application lifecycle. Now we’ll explore different options of implementing varying behavior depending on user type.

When an app has only one user type

I find it neat when an application allows me to use it before I’m forced to create an account or sign in. In such a case there will be a, potentially very long, time where we have no way of identifying the user. At the same time we’ll want to provide as much features of an identified user as possible. There will however be cases where application behavior will be different depending on whether the user is logged in or not.

A simple `if` will not do any harm, or will it?

Let’s assume that our application allows changing preferences for both anonymous and authenticated users. When saving or reading user dependent application settings we can write a simple if statement to decide which preferences to use.

Notice how in line 23 we do a check to decide the name of SharedPreferences to use. There’s nothing wrong with doing such a check in 1 or 2 places however this type of logic will typically spread over dozens and more places in the various parts of the code base.

The Null Object pattern to the rescue

Instead of relying on null checks we can use a simple pattern described in detail by Martin Fowler. Here’s how it would look like in our case:

Now all places where we require an easily substitutable information about the user can be handled with plain old polymorphism.

The above approach will work whenever we need a default value for the anonymous user - no so much when we need different behavior. It’s tempting to embed various actions into UserReference interface implementations (AnonymousUserReference and RegularUserReference) however this will make the mentioned interface depend on UI concepts which I would like to avoid.

When an app has more than one user type

Before I show how to avoid putting too much weight onto UserReference interface, let’s make our situation a bit more complicated. Let’s assume that our app can be used by three types of users: anonymous, regular and premium. A very common approach is to use an enum to handle that i.e.:

In fact because case supports default the above implementation works well as long as we have a small number of user types and there’s little shared behavior.

A slightly different approach is to hide the decision making and expose method(s) that accept different behaviors. The following code is an example of how we can approach encapsulating differences between user types in by explicitly declaring a behavior:

Note that the above uses double dispatch but it could easily be replaced with a switch statement. Using separate class to encapsulate differences should make it easier to keep unnecessary logic out of Activity.

A curious reader might notice that if we would like to change how we handle only one user type we’re still forced to override all methods. There’s a fairly simple way to improve the above though:

Now we can implement custom actions as well as indicate a default. As usual whether to go with a simple if statement, a switch or a double dispatch depends on a use case. However it’s useful to know the different ways of tackling the same problem.

This article is cross-posted with my company blog

Maintaining current user information in Android

Tue, 30 Jun 2015 00:00:00 GMT

Every but trivial android application needs to maintain information about current user - regardless if he has authenticated or not. While this may sound easy there are still at least handful of ways one can do it - in this article I’m going to explore couple of them.

Keeping state in custom `Application` class

This technique boils down to having custom application class like so:

While the above is both easy to write and understand there are several problems with it. First and foremost if you would like to get some user property you’ll need to reference MyApplication i.e.:

Since data about current user will be used in several places spread through the code each of those places will now be coupled to MyApplication. This on the other hand will jeopardize any attempt to break the code base into modules. Furthermore this approach may encourage sharing other global application state through static variables defined on custom Application class. In short don’t do this.

Keeping global state in `User` class

A slightly better idea is to move static variables from Application class into User class itself:

Now our Application class has one less responsibility and we don’t need smelly references spread around the codebase. We may also be able to split the project into modules - as long as we’re not too inclined to separating concerns. Problems arise as soon as you start unit testing though. Remember that good unit test should be as isolated as possible. Using global state through static methods on User class forces us to properly setUp() and tearDown() its state in every test. It would be much easier if we could properly inject information about current user through tested component(s) constructor(s).

I mentioned separation of concerns - does a user class have any reason to know whether a user has signed in? I don’t think so.

Separating concerns with User Session

I find it helpful to think about objects lifetimes when designing components. In every application that allows users to sign in there will be a time where we don’t have a user object to work with. To express that property I usually store the information about current user in a Session object. There is only one user session whenever an application is running. After user signs in the session is updated to represent it. After signing out it’s usually a good idea to reset session state. Consider following example:

Couple of things to note here. First and foremost the Session class is not public it’s local to package responsible for maintaining it’s state. Typically it’s a package dealing with user registration, sign in and keeping remote services auth tokens fresh. Secondly Session public interface only covers methods defined in CurrentUserInfo exposing methods that come handy when getting current user name. The Session object is not limited to one interface. Quite opposite it may implement more interfaces depending on different clients (components requesting various details about current user) needs.

Since Session embraces Interface Segregation Principle it’s much easier to test code that requests information about current user i.e AppBeahvior. A reusable fake can be coded with no effort - it only has to implement very limited set of methods.

Notice that I’ve not used static methods nor variables in above example. Instead I’ve simply marked the class with @Singleton annotation to ensure that there will only be single instance ever created, injected and used. I highly encourage using Dependency Injection frameworks (a.k.a IoC containers) with Dagger (and Dagger 2) taking the lead in Android space.

In the next article I’m going to show a way to deal with application behavior changing depending on user being logged in or not.

This article is cross-posted with my company blog

Anchor child element click

Sun, 25 Jan 2015 17:00:00 GMT

I had to solve a seemingly trivial bug in an angularjs based application that turned out to be more interesting than usual.

The bug

The bug report stated that “Clicking on a label causes page reload”. That should be an easy one I thought to myself and openeded chrome inspector to see a structure of DOM. Here’s a simplified version of markup:

<a  href="" ng-click="anchorAction($event)" ng-controller="ActionCtrl">
  Anchor
  <span ng-click="childAction($event)">
  A child
  </span>
</a>

module.controller('ActionCtrl', function($scope){
  $scope.anchorAction = function($event){
    console.log('anchorAction');
  };
  $scope.childAction = function($event){
    $event.stopPropagation();
    console.log('childAction');
  };
});

My intention was to have different behaviour when an anchor or a span element is clicked. Just as in the example above when a is clicked anchorAction should be printed whereas the same event triggered on span should only print childAction. Interestingly the actual behaviour is different.

When the anchor is clicked indeed a function attached by ng-click is executed properly. Note that even though we did not call $event.preventDefault() a page reload is not triggered. This is due to htmlAnchorDirective provided by angularjs which effectively prevents empty href attribute from taking action.

A click on span element will stop event from bubbling up document tree - thus preventing anchorAction from executing. In addition it will obviously print childAction and to my surprise it will cause a page reload.

Wait a second didn’t we just prevent the event from traveling up to the anchor element? Yes. So why does the page reload anyway?

Searching for a root cause.

Almost immediately I’ve verified that calling $event.preventDefault() inside childAction fixes the problem. The fix got checked in and will be deployed soon - case closed. I was unhappy though because I didn’t understand this behaviour at all.

At first I naively thought that it might be a Chrome bug - a quick check in FF and IE diminished this stupid idea.

Then I thought that it may be related to angularjs in some strange manner so I’ve prepared an example fiddle that demonstrates the issue. I’ve searched for and read many answers on Stack Overflow and other forums but none of them gave an in-depth explanation.

The HTML spec

Since the example fiddle demonstrated same behaviour in all major desktop browsers I realised that it must be part of HTML spec - after an hour or so it turned out that it was a good hunch.

Up until now I thought that an event (and a click event in particular) default action is dependent on an element it visits when it is dispatched through a DOM tree. In the above example it would mean that since I’ve stopped click from bubbling up it should not reach a element and because of that it should not execute its default action - in our case a page reload.

It turned out that my assumptions about events dispatching and elements default actions were wrong.

The relevant part of the specification describes activation behavior with an explanation of what I’ve experienced:

Let target be the element designated by the user (the target of event).

If target is a canvas element, run the canvas MouseEvent rerouting steps. If this changes event’s target, then let target be the new target.
Set the click in progress flag on target to true.
Let e be the nearest activatable element of target (defined below), if any.
If there is an element e, run pre-click activation steps on it.
Dispatch event (the required click event) at target.

If there is an element e and the click event is not canceled, run post-click activation steps on element e. If there is an element e and the event is canceled, run canceled activation steps on element e. 7. Set the click in progress flag on target to false.

The most relevant steps are 4. and 6. as they clearly indicate that target and nearest activatable element that triggers default action can be separate. What’s left to have a complete understanding is how nearest activatable element is defined:

Given an element target, the nearest activatable element is the element returned by the following algorithm:

If target has a defined activation behavior, then return target and abort these steps.

If target has a parent element, then set target to that parent element and return to the first step.

Otherwise, there is no nearest activatable element.

Now it is obvious why a default action of an anchor is executed even though a click event did not bubble up from its child.

This article is cross-posted with my company blog

Lazy Apk - a simple TeamCity artifact downloader

Tue, 06 Jan 2015 21:00:00 GMT

At Bright Inventions we use TeamCity as a continuous integration server. Apart from building, running tests and uploading artifacts we sometimes use it to quickly distribute an android application to clients and test team. However we found using TeamCity UI on a mobile device isn’t as pleasing as it could be. That’s why we usually recommend downloading updates through a dedicated application such as TeamCity Downloader.

While TeamCity Downloader is really easy to use I found it lacks a couple of features I need. First of all I would like to be able to use multiple TeamCity servers at once. Moreover it would be nice to see commit messages related to particular build to streamline change communication.

Lazy Apk - a simple apk downloader

Since I wanted to test some of new APIs available in Lollipop I thought it would be a good idea to try them out in an application that could actually be useful. That’s why I’ve build another artifacts downloader - Lazy Apk.

The tool allows you to configure multiple TeamCity build servers:

As I mentioned the commit messages are visible next to each build:

Downloaded APKs are stored inside android “Downloads” so there is no need to pull the same package twice:

Source available on Github

Lazy Apk source code is available on github. I’ve tried couple of different techniques to extract code from Activity classes - I’m not yet sure if I like the results though…

This article is cross-posted with my company blog

Google Play Services is no longer a giant monolith

Tue, 09 Dec 2014 10:00:00 GMT

Nowadays it’s getting harder and harder to build a meaningful app and not rely on Google Play Services to aid us in some commonly required features such as maps, better location provider, geo fencing and so much more. Unfortunately up until now the library shipped as a giant monolith ripping us from one third of dex method limit. For curious reader here’s are method counts in couple of versions:

Version	Method Count
3.2.65	6330
4.4.52	16933
5.0.89	20312
6.1.71	23641

and a full breakdown.

Google Play Services 6.5 granular dependency management

Today Google has made the awaited, more than usual, version of their SDK available. With the update apart from new features you can finally depend only on a subset of enormous API. Here’s a table from documentation along with dex method counts:

API Name	Gradle depdenency	Dex method count
Google Play Services	com.google.android.gms:play-services:6.5.87	24525
Google+	com.google.android.gms:play-services-plus:6.5.87	1525
Google Account Login	com.google.android.gms:play-services-identity:6.5.87	181
Google Activity Recognition	com.google.android.gms:play-services-location:6.5.87	857
Google App Indexing	com.google.android.gms:play-services-appindexing:6.5.87	482
Google Cast	com.google.android.gms:play-services-cast:6.5.87	976
Google Drive	com.google.android.gms:play-services-drive:6.5.87	2328
Google Fit	com.google.android.gms:play-services-fitness:6.5.87	1895
Google Maps	com.google.android.gms:play-services-maps:6.5.87	2568
Google Mobile Ads	com.google.android.gms:play-services-ads:6.5.87	3278
Google Panorama Viewer	com.google.android.gms:play-services-panorama:6.5.87	94
Google Play Game services	com.google.android.gms:play-services-games:6.5.87	5046
Google Wallet	com.google.android.gms:play-services-wallet:6.5.87	1116
Android Wear	com.google.android.gms:play-services-wearable:6.5.87	1187
Google Actions Google Analytics Google Cloud Messaging	com.google.android.gms:play-services-base:6.5.87	5212

A small change to improve build time

For me the biggest win is that in one of the apps we are actively developing granular dependency declaration means with a simple change from

compile 'com.google.android.gms:play-services:6.1.71'

compile 'com.google.android.gms:play-services-maps:6.5.87'
compile 'com.google.android.gms:play-services-location:6.5.87'
compile 'com.google.android.gms:play-services-base:6.5.87'

I no longer have to run Proguard during development. No wonder my build time just improved by 15 seconds.

This article is cross-posted with my company blog

Integrate slf4j with Crashlytics

Thu, 20 Nov 2014 21:00:00 GMT

As I mentioned in my previous post having meaningful log entries comes handy during development. When an app reaches beta testers as well as goes live it’s equally or even more important to be able to figure out why the app you’ve carefully coded isn’t behaving as it should. Testing the app on all android flavours is literally impossible that’s why getting an insight into what caused a crash is vital.

Crashlytics

Error reporting providers are getting more and more popular. There are plenty of options to choose from: Raygun, Airbrake and Crashlytics are just frew examples. At Bright Inventions we use the last one and are more and more pleased with it. Setting it up is really easy - if you don’t mind installing an IDE plugin it provides. Frankly I would prefer being able to configure a project with a simple command line tool but I understand a motivation behind it which is making the installation as seamless as possible.

Crashlytics let’s you not only report uncaught exceptions but also handled errors with additional information provided by log entries:

try {
  Crashlytics.log(Log.INFO, "HomeActivity", "Make request");
  makeRequest();
}catch(RuntimeException ex){
  Crashlytics.log(Log.ERROR, "HomeActivity", "Error making request " + ex);
}

This will print messages to logcat as well as well as well as make them available in Crashlytics dashboard.

Using slf4j with Crashlytics

I’ve already explained why I don’t like this approach to logging. Thankfully with slf4android it’s really easy to replace a default logcat appender with a CrashlyticsLoggerHandler:

public class CrashlyticsLoggerHandler extends Handler {
  MessageValueSupplier messageValueSupplier = new MessageValueSupplier();

  @Override
  public void publish(java.util.logging.LogRecord record) {
    LogRecord logRecord = pl.brightinventions.slf4android.LogRecord.fromRecord(record);
    StringBuilder messageBuilder = new StringBuilder();
    messageValueSupplier.append(logRecord, messageBuilder);
    String tag = record.getLoggerName();
    int androidLogLevel = logRecord.getLogLevel().getAndroidLevel();
    Crashlytics.log(androidLogLevel, tag, messageBuilder.toString());
  }

  @Override
  public void close() {}
  @Override
  public void flush() {}
}

All that is left is to add the handler to root logger in your custom application onCreate method:

LoggerConfiguration.configuration()
  .removeRootLogcatHandler()
  .addHandlerToRootLogger(new CrashlyticsLoggerHandler());

From now on all log messages will go through Crashlytics API and couple of last log entries will be available next to crash report details in dashboard.

This article is cross-posted my company blog

Introducing slf4android - a simple slf4j implementation for android

Sat, 01 Nov 2014 00:00:00 GMT

Every now and then you have a bug that is hard to reproduce or only happens on certain phones or android versions. The thing that really comes handy in such case is a detailed application log. That’s why it’s so important to take time to add useful log entries in every non trivial part of the codebase. At the very minimum you’ll want to log any errors.

Logging frameworks

That’s why it’s so important to create log entries easily. The default solution that comes with Android by means of Log is the most commonly used. However for me it’s really the least pleasant to use:

Log.e("MyTag", "Failed to download " + url +  "due to occurred " + ex);

I really don’t like that I have to specify a tag each time I need to log something. Moreover having to concatenate strings seems tedious and error prone.

Logging with `java.util.logging`

Another alternative that is available by default on android is packaged inside java.util.logging.*. And here’s an example of how to use it:

    private static final Logger LOG = Logger.getLogger(MyActivity.class.getName());
    // and then
    LOG.log(Level.FINE, "Starting activity {0} saved instance {1}", new Object[]{this, savedInstanceState});

You’re able to use MessageFormatter style to format log entries. However no variable arguments method overload makes it both harder to read and write. More importantly by default if you use above statement the message will not be printed anywhere. It’s easy to fix when you know where to look for.

Powerful `logback`

logback is probably the most powerful and configurable logging framework. Among many features you can for example send an email with 50 last log entries - I’ve used it in one project, it can be a bit hard to configure but it really comes handy during testing. In order to use it on android one needs to use a ported version logback-android. There is one caveat though - this library is costs about 512kB - and takes about 4200 out of dex method limit.

Simple android-logger

android-logger is a small (<50KB) library that let’s you use slf4j api to print to logcat. It ships with various configuration options that let you change the format of output messages as well as log level based on hierarchical logger names. However you won’t be able to print messages to an additional file and you can only configure the logger through properties files.

slf4android

Since I wasn’t perfectly happy with above and because some design decisions made in android-logger make it not so easy to add features like logging to a file, creating custom patterns and configuring it from code I decided to create yet another logging utility.

It’s a tiny wrapper around slf4j api baked by the java.util.logging logger mechanism. This means you can easily hook in any existing java.util.logging.Handler implementations.

To use this little gem you’ll need to add http://bright.github.io/maven-repo/ to repository list:

repositories {
    maven {
        url "http://bright.github.io/maven-repo/"
    }
}

and then declare a dependency inside a module:

dependencies {
    compile('pl.brightinventions:slf4android:0.0.4@aar'){
      transitive = true
    }
    //other dependencies
}

As with any slf4j compatible implementation using slf4android looks like this:

class HomeActivity extends Activity {
    private static final Logger LOG = LoggerFactory.getLogger(HomeActivity.class.getSimpleName());

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        LOG.debug("Hello from {} saved instance state {}", this, savedInstanceState);
    }
}

Logging to a file

To print messages to a separate file just add:

LoggerConfiguration.configuration().addHandlerToLogger("", LoggerConfiguration.fileLogHandler(this));

inside your custom android.app.Application onCreate method. This will create rotated log files inside context.getApplicationInfo().dataDir with a name derived from context.getPackageName() and a default message pattern %date %level [%thread] %name - %message%newline

More features

slf4android let’s you register custom message patterns and configure logging level - although the api for that is still rough around the edges. It also features a simple (and not well tested) mechanism for error reporting that, when enabled, will display a Dialog prompting tester to notify developer through email whenever an error is encountered. You can enable it with:

LoggerConfiguration.configuration().notifyDeveloperWithLogcatDataHandler(applicationContext, "developer.address@domain.com")

and receive emails with attached logcat output which comes handy during development.

Guide to javascript generators

Sun, 15 Jun 2014 00:00:00 GMT

For a while now I’ve been hearing and reading about a new exciting features of ES6 like arrow functions, array comprehensions, classes, modules, generators and many more. While many of them are easy to understand generators can take a while to grok. So I thought it would be a good idea to write some samples to master this particular feature.

What are generators

Generators or to be more precise generator functions are functions which can be exited and then reentered with function execution context (variable binding) preserved - a limited version of coroutines if you will. Traditionally they are used to write iterators more easily but of course they have many more interesting use cases.

To run following samples your node version has to support harmony generators. You can check that with node --v8-options | grep harmony. The latest unstable version 0.11.12 supports them.

A simple generator

Here’s a very simple example of generator that yields only two values: {% gist miensol/9cf18bc90b2021018403 %} when executed it will output:

{value: 1, done: false}
{value: 2, done: false}
{value: undefined, done: true}

A Fibonacci generator

Of course since we have an easy way to create iterators we have to have a way to iterate over them. One approach is to just use a while loop: {% gist miensol/367d27fb1b3f7966e31a %}

but honestly this would be a step back. Fortunately es6 comes with a new kind of for-of loop designed specifically to work with iterators:

for(var current of fibo()){
  console.log(current);
}

Iterators and iterables

I’ve mentioned couple of times a word iterator, but what exactly is that? In ES6 iterator is an object that has next method which when called returns and object having 2 properties:

value - which represents value returned by iterator
done - a boolean indicating whether iterator has done it’s job or not

As we saw in the above generator examples every generator is an implicit iterator thus we can loop through it using new for-of syntax.

There is one more object type it is good to know of - iterable. According to ES6 Wiki an object is iterable if has a internal property named iterator which is a function returning iterator. Unfortunatelly the latests specs aren’t yet implemented in v8 (or incorporated in node.js) so the following example is only my understanding of the specs:

var iterable = {};
iterable[Symbol.iterator] = function(){
    console.log('calling Symbol.iterator')
    return (function *(){
      yield 1;
      yield 2;
    }());
};

for(var item of iterable){
  console.log(item);
}

which would print:

calling Symbol.iterator
1
2

Passing values to generators

So far we’ve covered the most simplistic usage of generators for those familiar with .NET it reassembles a yield keyword introduced in C# 2.0 in 2005. Since ES6 Generators design was influenced a lot by python generators now wonder we can also pass values to generator functions. You do it by passing value to next function call: {% gist miensol/30aa3afe5396bdc6b8c1 %}

Running it should output something similar to:

How old are you? oracle says:  0.760754773626104
Why? oracle says:  0.36784305213950574
Thank you!

A careful reader will notice that the first question is silently ignored by our oracle generator. This is logical although might be strange at first. The generator function body isn’t executed until we call next for the first time - which is like starting the generator. The call to next will execute function body until it encounters yield - the control flow changes and the function is left to be entered later on - on future next call. Notice that we there is no logical place to store/assign the value passed to first next() call - a value that we pass to first next() call is ignored.

Thanks to @NOtherDev for pointing that out.

Generators error handling

Now that we know how to yield values and receive them inside generators. It’s equally important to be able to handle exceptions gracefully. Since generators are executing synchronously semantics of typical try {} catch {} should be preseved: {% gist miensol/f740e31ccc73283809a5 %} which would output:

Generator entered, yielding first value
Generator asked for second value - will go bum
[Error: You can only call this generator once, sorry!]

It’s important to note however that if you try to call next method on a generator that has thrown an exception you’ll get another error:

throwingGenerator.next();
                  ^
Error: Generator is already running
    at GeneratorFunctionPrototype.next (native)

Now this was a case when a generator has thrown exception at caller. What if we would like to tell generator that an error occured at caller’s site. Thanks to generator throw function that is easy enough:

{% gist miensol/d5dd0e6d46c05f51f0ed %}

This as you may expect will produce:

I will stop when you tell me about error
Got value from you: I will throw in a moment!
Got error from you: This is the end. Goodbye!

Practical generator usage

An obvious use case is of course creating lazy evaluated collections. One can now very easily implement collection operators that are available in LINQ. Consider following examples: {% gist miensol/e5c8f929e2fcdf0253b2 %} which when executed will output (note the order or DEBUG calls):

DEBUG: concat yielding first generator values
DEBUG: reverse will yield 6
DEBUG: filter predicate value true for item 6
DEBUG: take will yield  6 counter 1
DEBUG: select will yield Even number 6
Even number 6
DEBUG: reverse will yield 5
DEBUG: filter predicate value false for item 5
DEBUG: reverse will yield 4
DEBUG: filter predicate value true for item 4
DEBUG: take will yield  4 counter 2
DEBUG: select will yield Even number 4
Even number 4
DEBUG: concat yielding second generator values
At the end

Note the concat function implementation uses yield *otherGenerator() to yield values yielded by other generator which greatly reduces a boilerplate code needed otherwise.

Simplifying asynchronous operations

It may as first be a surprise that generators can streamline asynchronous code - how would synchronously executing yield help in managing asynchronous calls? If you think of the way we write asynchronous code in most of mainstream languages it typically boils down to either:

passing a callback function(s) that will be called when the operation completes
immediately getting back a promise of result from async function
using a library similar to Reactive Extensions - that when it comes to asynchronous code somewhat similar in usage to promises but offers very rich API

Now when you look at async code consuming:

var power = function(number, exponent){
  var deferred = Q.defer();
  setTimeout(function(){
    deferred.resolve(Math.pow(number, exponent))
  }, 300);
  return deferred.promise;
};

power(2,3).then(function(result){
  console.log(result);
});

I immediately translate promise calls in my head to a more natural flow:

var result = _wait_for_async_ power(2,3);
console.log(result);

Now since we don’t have async feature like the one C# 5 has (or rather we don’t yet have it) it’s not yet possible to achieve this kind of simplification. Still if you think of how the imaginary _wait_for_async_ keyword would work it seems that it (in case of promise based implememntation):

would wait for the promise to complete in a non blocking fashion - possibly leaving function execution context
when the promise completes it would return to the exact place from where it was called and return value that it got from promise or throw an exception

Essentially that’s the way yield keyword works - however we still need an umbrella machinery that will take care of calling generator next method so it will return to the original caller as well as it throw method to report exceptions.

As you may have guessed there are already plenty of implementations of this machinery - the one I like most is co. Take a look at an example from their page and enjoy callback free life!

Setting up fluentd elasticsearch and Kibana on azure

Sun, 08 Jun 2014 00:00:00 GMT

A decent way too go through log files can come handy in various ways. From debugging bugs, investigating problems on production environment to figuring out where performance bottle necks are. For simple cases when there is only one log file involved you can go very far with using grep/awk/sed magic or any decent editor capable of handling regex in large files.

It becomes tricky however when your logs are scattered through several files, maybe use different log format or stamp entries with different timezones. Fortunately there are services like Splunk, Loggly and many others that take away great amount of pain involved in aggregating distributed log entries and searching through them.

I was searching once for a free alternative that I could quickly get up and running in our test environment where we had 10+ machines running 20+ web sites or services that we’re all part of a single product. That’s when I found out about Kibana, logstash and how they make use of elasticsearch.

Back then I considered using fluentd instead of logstash however most of the machines we were running windows and that is not where fluentd shines. Nowadays I’m using linux vm mostly that’s why I decided to use fluentd.

Setup a vm for elasticsearch

We need a machine that will store our aggregated log entries so let’s create one using Azure Cross-Platform Command-Line interface. After installing it with npm install azure-cli -g we need to introduce ourself to it. The simplest way to do this is to issue azure account download which will open a browser to download your profile: After that you’ll need to import the downloaded file with npm account import /path/to/your/credentials.publishsettings.

I’m used to ubuntu so let’s find an machine image to use:

azure vm image list | grep Ubuntu-14
data:    b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140122.1-alpha2-en-us-30GB                          Public    Linux  
data:    b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140226.1-beta1-en-us-30GB                           Public    Linux  
data:    b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140326-beta2-en-us-30GB                             Public    Linux  
data:    b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140414-en-us-30GB                                   Public    Linux  
data:    b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140416.1-en-us-30GB                                 Public    Linux  
data:    b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140528-en-us-30GB                                   Public    Linux

The last one looks like most recent so we’ll used that. Next we need to create a virtual machine. We can use either password authentication (which isn’t recommended) or a ssh certificate. To use the latter we need to create a certificate key pair first. Openssl command line tool will do it for us:

openssl req -x509 -nodes -days 730 -newkey rsa:2048 -keyout ~/.ssh/mymachine.key -out ~/.ssh/mymachine.pem

It’s now time to issue azure to create a new instance of Ubuntu machine for us:

azure vm create machine-dns-name b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140528-en-us-30GB machineuser \
  --vm-size extrasmall \
  --location 'North Europe' \
  --ssh 22 --no-ssh-password --ssh-cert ~/.ssh/mymachine.pem

info:    Executing command vm create
+ Looking up image b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140528-en-us-30GB
+ Looking up cloud service
+ Creating cloud service
+ Retrieving storage accounts
+ Configuring certificate
+ Creating VM
info:    vm create command OK

We can find out more details about our newly created machine with: azure vm list --dns-name machine-dns-name --json. Our machine is up and running to so we can log into it using ssh (before you do that be sure to change .key file permission to 0600 so that only you can access it): ssh -i ~/.ssh/mymachine.key machineuser@machine-dns-name.cloudapp.net

Install elasticsearch

First let’s download elasticsearch and extract it to our newly created vm:

curl https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.2.0.zip -o elasticsearch-1.2.0.zip
unzip elasticsearch-1.2.0.zip

To run elasticsearch we’ll need java in version at least 1.7:

sudo apt-get update
sudo apt-get install default-jre

Now we can finally run elasticsearch:

./elasticsearch-1.2.0/bin/elasticsearch

If we want to run elasticsearch as a daemon we can pass -d to the script. (see below for init.d script)

We also need to open port for elasticsearch:

azure vm endpoint create --endpoint-name elasticsearch machine-dns-name 9200 9200

We only wan’t our machines to be able to push logs to elasticsearch that’s why we need to add ACL rules to the endpoint we created above. Unfortunately I couldn’t find a way to do this through CLI interface so we need to resort to web interface.

Install Kibana

Kibana is a web application designed to perform elasticsearch searches and display them using neat interface. It communicates with elasticsearch directly that’s why it’s not really suited to be deployed to publicly accessible place. Fortunately there is kibana-proxy which provides authentication on top of it. To run kibana-proxy we’ll need node.js let’s install both of them:

sudo apt-get update
sudo apt-get install git
sudo apt-get install nodejs
sudo apt-get install npm

git clone https://github.com/hmalphettes/kibana-proxy.git
cd kibana-proxy
git submodule init
git submodule update
npm install
node app.js &

kibana-proxy is now running but in order to access it from outside we need to open firewall port:

azure vm endpoint create --endpoint-name kibana-proxy machine-dns-name 80 3003

Now when you point your browser to http://machine-dns-name/ you should see kibana welcome screen.

Restrict access to kibana-proxy

You probably don’t want anyone to be able to see your logs that’s why we need to setup authentication. kibana-proxy implements this through Google OAuth - so let’s enable it for our installation. First we’ll need APP_ID (Client ID) and APP_SECRET (Client secret) so let’s go to google developer console and create a new project. Next we need to enable OAuth for our project in APIs & auth > Credentials:

Make sure AUTHORIZED REDIRECT URI looks like http://machind-dns-name.cloudapp.net/auth/google/callback. For more secure setup you should be using https but for simplicity we’ll skip that.

Now we need to start kibana-proxy again this time passing it APP_ID, APP_SECRET along with AUTHORIZED_EMAILS through environment variables (kibana-proxy-start.sh):

export APP_ID=fill_me_appid
export APP_SECRET=fill_me_app_secret
export AUTHORIZED_EMAILS=my_email_list
exec /usr/bin/nodejs app.js

Now when you go to `http://machine-dns-name/` you should get redirect to Google authorization page.

We want to run the kibana-proxy continuously - the simplest way to achieve that is to make use of forever module. However in this guide we’ll use upstart scripts described below.

Fluentd installation

To push log entries to our elasticsearch instance we need fluentd agent installed on the machine we want to gather logs from. Fluentd agent comes in 2 forms: fluentd gem if you wish to have more control features and updates and td-agent if you care for stability over new features. We’ll go with fluentd gem for this guide.

Since fluentd is implemented in Ruby with most performance critical parts written in C. To install it we need ruby first but obtaining it with rvm is a piece of cake:

curl -sSL https://get.rvm.io | bash -s stable
source ~/.rvm/scripts/rvm
rvm install 2.1.1
sudo gem install fluentd --no-ri --no-rdoc
sudo fluentd -s

Now we have fluentd installed with sample configuration available in /etc/fluentd/fluent.conf.

Since we would like our log entries to be pushed to elasticsearch we need a plugin for that too:

gem install fluent-plugin-elasticsearch
gem install fluent-plugin-tail-multiline

Let’s edit fluent.confg file so that our agents reads sample log files from rails app and forwards them to elasticsearch instance:

<source>
  type tail_multiline
  format /^(?<level_short>[^,]*), \[(?<time>[^\s]*) .(?<request_id>\d+)\]\s+(?<level>[A-Z]+)[-\s:]*(?<message>.*)/
  path /path/to/log/file/staging.log
  pos_file /path/to/positions/file/fluentd_tail.pos
  tag app_name.environment
</source>

<match app_name.**>
  type elasticsearch
  host machine-dns-name.cloudapp.net
  logstash_format true
  flush_interval 10s # for testing
</match>

The format we see above will following line of Rails log file:

I, [2014-05-22T12:24:44.434963 #11837]  INFO -- : Completed 200 OK in 884ms (Views: 395.7ms | ActiveRecord: 115.0ms)

and break it down to following groups/fields:

Key	Value
time	2014/05/22 12:24:44
level_short	I
request_id	11837
level	INFO
message	Completed 200 OK in 884ms (Views: 395.7ms \| ActiveRecord: 115.0ms)

Note that although builtin in_tail plugin supports multiline log entries I either don’t completely understand how to use it or there is no way to have full stack trace logged as a single message. Moreover tail_multiline doesn’t seem to support multiple input paths that’s why in order to monitor multiple files you would have to duplicate the source section.

The only thing left is to start fluentd and watch our logs

rvmsudo fluetnd -d

Upstart for fluentd, elasticsearch and kibana-proxy

The final thing we need to do is to setup above components to be run as services. Since we’re using Ubuntu for our servers we’ll use upstart congiurations for that.

Upstart for fluentd

Since we have installed fluentd in ubuntu user rvm but we wan’t we have to generate a wrapper script for it

rvm wrapper default fluentd

and then use generated wrapper ~/.rvm/wrappers/default/fluentd in our upstart script. Let’s create our upstart script and put it inside /etc/init/fluetnd.conf

description "Fluentd"
start on runlevel [2345] # All except below
stop on runlevel [016] # Halt, Single-User, Reboot
respawn
respawn limit 5 60 # Restart if ended abruptly
exec sudo -u ubuntu /home/ubuntu/.rvm/wrappers/default/fluentd

Upstart for elasticsearch

Elasticsearch already comes with a script for starting it in properly configured jvm we’ll utilize it in our upstart config /etc/init/elasticsearch.conf

description "Elasticsearch"
start on runlevel [2345] # All except below
stop on runlevel [016] # Halt, Single-User, Reboot
respawn
respawn limit 5 60 # Restart if ended abruptly
setuid ubuntu
exec /path/to/your/elasticsearch/bin/elasticsearch

Upstart for kibana-proy

With the kibana-proxy-start.sh the upstart configuration for kibana-proxy is dead simple:

description "kibana-proxy"
start on runlevel [2345] # All except below
stop on runlevel [016] # Halt, Single-User, Reboot
respawn
respawn limit 5 60 # Restart if ended abruptly
chdir /where/you/installed/kibana-proxy
setuid ubuntu
exec ./kibana-proxy-start.sh

Running services

Once you have your upstart scripts configuration ready you can now start services with a descriptive syntax:

sudo service elasticsearch start

I you’re having trouble getting upstart to work you can always check its configuration with:

initctl check-config

To find out why a particular service won’t start check out

dmesg | grep elasticsearch

Finally you can find you log files in /var/log/upstart/servicename.*

Now if you have setup everything correctly and of course you have some log messages generated by apps when you navigate to http://machine-dns-name.cloudapp.net/index.html#/dashboard/file/logstash.json you should see a nice dashboard filling up with log entries:

ng-click and double post

Wed, 21 May 2014 00:00:00 GMT

Recently I was fixing a bug in an application written in angularjs which turned out to be pretty interesting for me. At first I couldn’t reproduce the error and just when I got frustrated it happened exactly as stated in the issue tracker. As it turned out the problem was caused by fast clicking on the same button. Click handler triggered an http POST to server which creates a row in table - no wonder it looked strange when 2 rows where created after someone accidentally clicked twice.

Some solutions

I’ve quickly googled to find out what others are doing in such scenario and found at least couple of ways of handling it - some of them I have considered while others where completely new to me.

Disable button

The most simple approach is to use ng-disabled on the same button as ng-click directive. Which would look like this:

<button ng-click="executePost()" ng-disabled="buttonDisabled">Click me</button>

$scope.executePost = function(){
  $scope.buttonDisabled = true;
  runHttpPost().then(function(){
    //do something useful with a response
  }).finally(function(){
      $scope.buttonDisabled = false;
  })
};

As you can see the code is pretty straight forward especially when you make use of angularjs promise api provided by $q service.

Stop relying on the result of asynchronous action

Another approach is to design user interface in such a way that the we can safely predict what result the action will produce. For instance if we’re editing an entity and have performed as much validation as possible on client side it’s offen safe to assume that the request will be processed successfully and our changes will persist. If so we don’t have to wait for the server to finish processing and can move to a next screen in the application that will display values already available in browser. As a bonus our application will feel much faster and in many cases it makes a significant difference. It’s not always easy though and sometimes requires significant changes in the way we design our user interface as well as our server api. Alex MacCaw summarised it nicely.

Make sure a second request will not make it to the server

An approach I haven’t thought about is to actually prevent second request that looks identical to the first one and is created just after the first one. Michał Ostruszka described an implememtation of this technique. I haven’t actually tried it as it seemed way too complex for an issue that I was fixing. The approach can be very tempting as it allows to handle double requests globally - once per application. Although I suspect it may be difficult to use if the act of making request that was blocked is causing side affects to the state of the application.

`bi-click` approach

Most of async operation in modern browser based libraries are expressed using a flavour of promise/deferred api. This is no different in angularjs which provides $q. So if we would like to block the button until the operation is completed it seems like a natural fit to use promises to achieve that. Here is an example of such directive:

Further usage

Every web developer out there has probably implemented an ajax throbber of some kind. In Angularjs world most of the examples I see are very similar to our button blocking with isDisabled flag. Usually it will mean something along these lines:

$scope.executePost = function(){
  $scope.showThrobber = true;
  runHttpPost().finally(function(){
      $scope.showThrobber = false;
  })
};

<div class="ajax-thorbber" ng-show="showThrobber"></div>

While there is nothing wrong in particular with this example I find handling those flags in a controller rather tedious. As with the bi-click example one can very easily create a directive to handle throbber toggling in one place.

Nonintrusive http proxy in nodejs

Sun, 23 Mar 2014 22:14:00 GMT

In my previous posts I wrote about problems that might occur when using http proxy written in nodejs to debug http issues. Today I’m going to describe how to use nodejs builtin parser to overcome these problems. ##Nodejs streams Node.js has decent support for handling streams. Especially the pipe function takes a lot of burden away - whereas previously it had to be carefully handled by programmer. As a first step let’s use a simple socket server and tunnel http requests through it:

var net = require('net');
var stdout = process.stdout;
var server = net.createServer(function(clientSocket){
  var serverSocket = net.connect(8888,'localhost', function(){

      clientSocket.pipe(serverSocket);
      clientSocket.pipe(stdout);

      serverSocket.pipe(clientSocket);
      serverSocket.pipe(stdout);

  });
});
server.listen(9000);

The above code creates a socket server listening on port 9000. When it gets a connection from client it will immediately try to connect to other server listening on port 8888 - this is a port used by Fiddler or Charles. After the connection is established it uses pipe function to pass all data coming in on port 9000 to port 8888 as well as to standard output just so we can see what data is sent from client. We also need to pass data returned from port 8888 back to client that’s why there is the second pair of pipe calls.

With that we can already use curl to see raw http traffic written directly to standard output.

curl http://wp.pl/ --proxy http://localhost:9000/

##Creating http proxy on network level The above example albeit simple doesn’t really provide any value as we still need a external proxy to properly pass http traffic. In order to make it a real http proxy we need to add parsing logic that will extract information about target server where we should pass incoming data too. While building a very simple http parser isn’t difficult why don’t we use existing one that is built right into node.js core http module?

Using node.js http parser isn’t exactly documented and I guess it’s not part of public API. Nevertheless it is exposed to client code intentionally and a quick look through the core http module source code gives enough examples of how to use it. With existing parser instance the only thing left to do is to extract Host header value, use it to connect to target server. Here is a core part of code required:

net.createServer(function (socketRequest) {
    var requestParser = HttpParsingStream.createForRequest(socketRequest);

    requestParser.on('headers', function (req) {
        //pause the request until we setup necessary plumbing
        req.pause();
        //extract information about target server
        var hostNameHeader = req.headers.host,
            hostAndPort = hostNameHeader.split(':'),
            host = hostAndPort[0],
            port = parseInt(hostAndPort[1]) || 80;
        // now we now where to tunnel client request
        var srvSocket = net.connect(port, host, function () {
            //a response parser as a replacement for stdout
            var responseParser = HttpParsingStream.createForResponse(srvSocket);
            srvSocket.pipe(responseParser);
            //pipe data from server to client
            srvSocket.pipe(socketRequest);

            //flush data buffered in parser to target server
            requestParser.writeCurrentChunks(srvSocket);
            //pipe remaining data from client to target server
            socketRequest.pipe(srvSocket);
            //resume processing
            req.resume();
        });
    });

    //pipe data from client to request parser
    socketRequest.pipe(requestParser);

}).listen(9000);

The above code makes use of HttpParsingStream which is a hand rolled writable Stream that uses node.js http parser to emit events. As you can see we first pipe client socket to requestParser to get information about target server. As soon as we get headers event the incoming client request is paused, we setup connection to target server, write raw data buffered in requestParser and setup pipes in a similar fashion as in the previous example. The most important property of this http proxy is that it does not change data coming from client and from target server in any way which is invaluable when debugging problems in http implementations.

HttpParsingStream explained

The above example relies on 2 instances of HttpParsingStream for request and response respectively:

var net = require('net'),
    http = require('http'),
    stream = require('stream'),
    Writable = stream.Writable,
    parsers = http.parsers,
    HTTPParser = process.binding('http_parser').HTTPParser,
    util = require('util');

var HttpParsingStream = function (options) {
    Writable.call(this, {});
    var socket = options.socket;
    //get an instance of node parser
    var parser = parsers.alloc();
    //buffer for raw data
    var streamChunks = [];
    var that = this;
    //initialize as request or response parser
    parser.reinitialize(options.parserMode);
    parser.socket = socket;
    socket.parser = parser;
    //called by node http module when headers are parsed
    parser.onIncoming = function (request) {
        that.emit('headers', request);
        request.on('data', function () {
            //this is one of ways to get 'end' event
        });
        request.on('end', function () {
            that.emit('end');
            //free parser
            freeParser(parser, request);
        });
    };
    socket.on('close', function () {
        that.emit('socket.close');
    });

    this._write = function (chunk, encoding, callback) {
        streamChunks.push({
            chunk: chunk,
            encoding: encoding
        });
        //pass data to parser
        parser.execute(chunk, 0, chunk.length);
        callback && callback();
    };

    //write data currently in buffer to other stream
    this.writeCurrentChunks = function (writableStream) {
        streamChunks.forEach(function (chunkObj) {
            writableStream.write(chunkObj.chunk, chunkObj.encoding);
        });
    };
};
util.inherits(HttpParsingStream, Writable);

The HttpParsingStream accepts 2 options:

socket for underlying request and response
parserMode used to properly initialise node.js http parser

Here’s how we create objects with it:


HttpParsingStream.createForRequest = function (socket) {
    return new HttpParsingStream({
        socket: socket,
        parserMode: HTTPParser.REQUEST,
        //used only for debugging
        name: 'request'
    });
};
HttpParsingStream.createForResponse = function (socket) {
    return new HttpParsingStream({
        socket: socket,
        parserMode: HTTPParser.RESPONSE,
        //used only for debugging
        name: 'response'
    });
};

Because HttpParsingStream is a Writable stream we can use it as:

socketRequest.pipe(HttpParsingStream.createForRequest(socketRequest));

serverSocket.pipe(HttpParsingStream.createForResponse(serverSocket));

and let node.js code handle buffering, pausing and resuming. There is also one additional function used to clean up and return a parser instance back to pool - it’s a copy paste from node.js http module source code:

function freeParser(parser, req) {
    if (parser) {
        parser._headers = [];
        parser.onIncoming = null;
        if (parser.socket) {
            parser.socket.onend = null;
            parser.socket.ondata = null;
            parser.socket.parser = null;
        }
        parser.socket = null;
        parser.incoming = null;
        parsers.free(parser);
        parser = null;
    }
    if (req) {
        req.parser = null;
    }
}

Why create yet another http proxy implementation?

proxy-mirror a simple http debugging tool that I wrote so far relies on an excellent http-proxy module. However because http-proxy was created to be used mostly as a reverse proxy to route http traffic to different http servers it does not provide a way to access raw tcp data. With the above code I now have an easy way to access the data - so I can display it in a byte level manner - a feature of Fiddler that I find handy from time to time.

Moreover the target server will receive a request from client in an unchanged form. The same applies for response received by client. This is extremely important when resolving issues related to improper implementations of HTTP protocol.

I haven’t yet checked how the above code handles WebSocket connections - I’ll explore that in a next post. For the referece you can find full code in this gist.

Curl and a missing slash

Mon, 24 Feb 2014 21:55:00 GMT

While I was playing around with proxy-mirror I noticed an interesting behaviour when testing the proxy with curl. The following command:

curl http://wp.pl --proxy http://localhost:8888/

will result in following output when the proxy is Fiddler:

[Fiddler] Response Header parsing failed.
Response Data:
<plaintext>
43 6F 6E 6E 65 63 74 69 6F 6E 3A 20 63 6C 6F 73 65 0D 0A 0D 0A 3C 48 54  Connection: close....<HT
4D 4C 3E 3C 48 45 41 44 3E 3C 54 49 54 4C 45 3E 34 30 30 20 42 61 64 20  ML><HEAD><TITLE>400 Bad
52 65 71 75 65 73 74 3C 2F 54 49 54 4C 45 3E 3C 2F 48 45 41 44 3E 0A 3C  Request</TITLE></HEAD>.<
42 4F 44 59 3E 3C 48 32 3E 34 30 30 20 42 61 64 20 52 65 71 75 65 73 74  BODY><H2>400 Bad Request
3C 2F 48 32 3E 0A 59 6F 75 72 20 72 65 71 75 65 73 74 20 68 61 73 20 62  </H2>.Your request has b
61 64 20 73 79 6E 74 61 78 20 6F 72 20 69 73 20 69 6E 68 65 72 65 6E 74  ad syntax or is inherent
6C 79 20 69 6D 70 6F 73 73 69 62 6C 65 20 74 6F 20 73 61 74 69 73 66 79  ly impossible to satisfy
2E 0A 3C 48 52 3E 0A 3C 41 44 44 52 45 53 53 3E 3C 41 20 48 52 45 46 3D  ..<HR>.<ADDRESS><A HREF=
22 68 74 74 70 3A 2F 2F 77 77 77 2E 77 70 2E 70 6C 2F 22 3E 61 72 69 73  "http://www.wp.pl/">aris
3C 2F 41 3E 3C 2F 41 44 44 52 45 53 53 3E 0A 3C 2F 42 4F 44 59 3E 3C 2F  </A></ADDRESS>.</BODY></
48 54 4D 4C 3E 0A                                                        HTML>.

while a simple implementation relaying on node.js core http module and http-proxy module outputs this:

An error has occurred: {"bytesParsed":0,"code":"HPE_INVALID_CONSTANT"}

Meanwhile without the proxy parameter the actual response is:

HTTP/1.1 301 Moved Permanently
Server: aris
Location: http://www.wp.pl
Content-type: text/html
Content-Length: 0
Connection: close

Curl forgiving behaviour

As it turns out the actual outgoing HTTP request is different depending on the presence of –-proxy parameter. Without it the target server receives and responds with:

GET / HTTP/1.1
User-Agent: curl/7.26.0
Host: wp.pl
Accept: */*

HTTP/1.1 301 Moved Permanently
Server: aris
Location: http://www.wp.pl
Content-type: text/html
Content-Length: 0
Connection: close

but when the proxy setting is present:

GET http://wp.pl HTTP/1.1
User-Agent: curl/7.26.0
Host: wp.pl
Accept: */*
Proxy-Connection: Keep-Alive

UNKNOWN 400 Bad Request
Server: aris
Content-Type: text/html
Date: Sun, 23 Feb 2014 16:01:36 GMT
Last-Modified: Sun, 23 Feb 2014 16:01:36 GMT
Accept-Ranges: bytes
Connection: close

<HTML><HEAD><TITLE>400 Bad Request</TITLE></HEAD>
<BODY><H2>400 Bad Request</H2>
Your request has bad syntax or is inherently impossible to satisfy.
<HR>
<ADDRESS><A HREF="http://www.wp.pl/">aris</A></ADDRESS>
</BODY></HTML>

As you may have noticed the difference in requests (apart from additional header) is in first line – where in the first case curl assumed we want to GET /. The former uses a relative URI while the later absolute. Now if we change the command line just a little bit so that the address looks like http://wp.pl/ the server will receive request with correct absolute URI and will respond with 301.

UNKNOWN Status Line

Careful readers may have already noticed the Status Line of response returned by server is malformed. According to the spec it should have following form:

Status-Line = HTTP-Version SP Status-Code SP Reason-Phrase CRLF

That’s the reason why Fiddler warns users about protocol violation and the reason of error inside node.js proxy.

A less intrusive proxy

An example described above leads to other questions about behaviour of http proxy especially if we think about implementing HTTP debugging tool like proxy-mirror. It’s only a guess but I suspect that there are other subtle differences between requests sent by client and those received by target server when the proxy is implemented using http-proxy module or core http module. I am aware of at least one such case where HTTP Headers are lowercased. If I’m right a debugging tool relaying on them would make it really hard to spot certain problems – like protocol violations on both client and server side.

In my next blog post I’ll describe how to built even less intrusive proxy without building the http parser by hand.

An HTTPS system proxy with node

Fri, 03 Jan 2014 06:34:00 GMT

Creating a simple http proxy in node.js is super easy thanks to an excellent module – http-proxy. With only the following code you’ll have a proxy that can be used as system or browser level proxy:

``` var httpProxy = require('http-proxy'); var proxyServer = httpProxy.createServer(function (req,res,proxy) { var hostNameHeader = req.headers.host, hostAndPort = hostNameHeader.split(':'), host = hostAndPort[0], port = parseInt(hostAndPort[1]) || 80; proxy.proxyRequest(req,res, { host: host, port: port }); }); proxyServer.listen(8888); ```

Adding support for HTTPS

The first thing we will need is a certificate that will be used by TLS implementation for encryption. A server has access to certificate private and public key thus it is able to get a clear text from a message encrypted with public key. It’s a common knowledge that asymmetric encryption is more expensive in terms of CPU cycles than its symmetric counterpart. That’s way when using HTTPS connection asymmetric encryption is only used during initial handshake to exchange a session key in a secure manner between client and server. This session key is then used by both client and server for traffic encryption using symmetric algorithm.

Naturally to get or rather to buy a proper certificate one would have to be verified by a valid certification authority. Fortunately for the sake of a demo we can generate a self signed certificate. That’s really easy, a nice description is available on heroku help pages:

``` openssl genrsa -des3 -passout pass:x -out proxy-mirror.pass.key 2048 echo "Generated proxy-mirror.pass.key"

openssl rsa -passin pass:x -in proxy-mirror.pass.key -out proxy-mirror.key rm proxy-mirror.pass.key echo “Generated proxy-mirror.key”

openssl req -new -batch -key proxy-mirror.key -out proxy-mirror.csr -subj /CN=proxy-mirror/emailAddress=piotr.mionskowski@gmail.com/OU=proxy-mirror/C=PL/O=proxy-mirror echo “Generated proxy-mirror.csr”

openssl x509 -req -days 365 -in proxy-mirror.csr -signkey proxy-mirror.key -out proxy-mirror.crt echo “Generated proxy-mirror.crt”


<h3>http-proxy support for https</h3>
<p>http-proxy module supports various https configuration for example passing traffic from https to http and vice versa. Unfortunately I couldn’t find a way to get it working without preconfiguring target host and port – which was a problem while I was implementing
  <a
  href="https://github.com/miensol/proxy-mirror" target="_blank">proxy-mirror</a>. Here is what <em>curl</em>will print out when trying to use such proxy:</p>

curl -vk —proxy https://localhost:8888/ https://pl-pl.facebook.com/

timeout on name lookup is not supported
About to connect() to proxy localhost port 8888 (#0)
Trying 127.0.0.1…
connected
Connected to localhost (127.0.0.1) port 8888 (#0)
Establish HTTP proxy tunnel to pl-pl.facebook.com:443

> CONNECT pl-pl.facebook.com:443 HTTP/1.1 > Host: pl-pl.facebook.com:443 > User-Agent: curl/7.26.0 > Proxy-Connection: Keep-Alive >


<p>I suspect the problem is inherently related to the way http-proxy uses nodejs core http(s) modules – I might be wrong here though. The workaround I’ve used in <a href="https://github.com/miensol/proxy-mirror" target="_blank">proxy-mirror</a> was to listen
  to <em><a href="http://nodejs.org/api/http.html#http_event_connect" target="_blank">CONNECT</a>    </em>event on http server, establish socket connection to a fake https server listening on different port. When the https connection is established the
  fake https server handler proxies requests further. The advantage of this approach is that we can use the same proxy address for both http and https. Here is the code:</p>

var httpProxy = require(‘http-proxy’), fs = require(‘fs’), https = require(‘https’), net = require(‘net’), httpsOptions = { key: fs.readFileSync(‘proxy-mirror.key’, ‘utf8’), cert: fs.readFileSync(‘proxy-mirror.crt’, ‘utf8’) };

var proxyServer = httpProxy.createServer(function (req, res, proxy) { console.log(‘will proxy request’, req.url); var hostNameHeader = req.headers.host, hostAndPort = hostNameHeader.split(’:’), host = hostAndPort[0], port = parseInt(hostAndPort[1]) || 80; proxy.proxyRequest(req, res, { host: host, port: port }); });

proxyServer.addListener(‘connect’, function (request, socketRequest, bodyhead) { var srvSocket = net.connect(8889, ‘localhost’, function () { socketRequest.write(‘HTTP/1.1 200 Connection Established\r\n\r\n’); srvSocket.write(bodyhead); srvSocket.pipe(socketRequest); socketRequest.pipe(srvSocket); }); });

var fakeHttps = https.createServer(httpsOptions, function (req, res) { var hostNameHeader = req.headers.host, hostAndPort = hostNameHeader.split(’:’), host = hostAndPort[0], port = parseInt(hostAndPort[1]) || 443;

proxyServer.proxy.proxyRequest(req, res, { host: host, port: port, changeOrigin: true, target: { https: true } }); });

proxyServer.listen(8888); fakeHttps.listen(8889);


<h2>HTML5 WebSocket support</h2>
<p>The above code has still problems handling <a href="http://en.wikipedia.org/wiki/WebSocket" target="_blank">WebSockets</a>. This is because browsers, according to the spec, change the way they establish initial connection when they detect that they are
  behind an http proxy. As you can read on <a href="http://en.wikipedia.org/wiki/WebSocket#Proxy_traversal" target="_blank">wikipedia</a> more existing http proxy implementation suffer from this.&#160; I still haven’t figured out how to handle this scenario
  elegantly with http-proxy and node.js, when I do I will post my findings here.</p>

Building an HTTP sniffer with node.js

Wed, 01 Jan 2014 22:14:00 GMT

Whether to inspect a server response, optimize network usage or just to fiddle with new REST API it almost always make sense to use an application that can display http requests and responses for investigation. On Windows there is the great Fiddler. While it’s possible to use Fiddler on other platforms using virtualization software I think it’s an overkill. Fortunately there are alternatives. Charles looks like the most advanced one offering most if not all features available in Fiddler. There is also a little less popular HTTP Scoop. Both Charles and HTTP Scoop aren’t free but in my opinion they are worth the price especially if used often. Command line lovers might find mitmproxy suit their needs. If you only need basic features ngrok might serve you well. To dive a bit deeper and see http traffic on a tcp level WireShark is indispensable.

As you can see there are plenty of tools available to help understand what is happening on http level. As I was learning about http caching a question popped to my head. How hard would it be to actually build a simple http sniffer?

proxy-mirror – a simple http inspector

As it turned out it’s actually not that hard to built one thus proxy-mirror came to be. Of course this is a very simplistic http sniffer – nowhere near to tools I mentioned above both in terms of features and reliability – but it works (at least in most scenarios :-)). It’s open source and I learned couple of new things about HTTP while implementing it – more on that in future posts. Here are some screen shoots of the the tool:

As you might have guessed from screenshots proxy-mirror is a web application. Right now it’s not very easy to try it out – the instructions are in the Readme – I’ll try to fix it soon.

How it works

I wanted to run proxy-mirror on many platforms and rely on a foundation that has both great support for http and building web applications. I picked node.js over java or ruby mainly because I wanted to sharpen my skills in it.

You can think of proxy-mirror as 2 logical components. The first is a regular http proxy that you can use by configuring your browser or system. I didn’t want to focus on building that first so I utilized a great node module call http-proxy. With it’s helped you can have a system wide proxy in couple of minutes. The http proxy component emits events whenever a request or response goes through it. Those events are consumed by the second logical component – a web application built with express.js. The information about http traffic received from events is then pushed to browser part through socket.io where a simple SPA built with my beloved AngularJS displays information about them.

Features

Right now proxy-mirror has only couple of features:

http and https support – although the latter one requires additional setup
simple session list – a grid with list of request/response pairs that you can inspect
a detailed view – both for request and response that can display headers, message body and a preview right in the browser

Although it still is a very simple (and buggy) application I actually found that it has most of features I frequently use – probably except for filtering. Maybe someday it will become a reasonable alternative for Charles - at least for me.

I think building an http sniffer is a great exercise in the process of learning how http(s) works. I guess the famous saying about journey being more more important than the destination fits nice here.

Combine Express.js and ASP.NET Web API

Sun, 03 Nov 2013 12:00:00 GMT

I’m really excited about the OWIN initiative. This “standard” enables countless new ways of combining existing http web stacks with .net – a thing that up until recently was either impossible or not feasible. It gets even more positive knowing that Microsoft supports the case actively.

OWIN - why you should care?

Open Web Interface for .NET aims to define a standard interface between web servers and web applications. Adhering to this interface will allow decoupling of server implementations from web frameworks. The idea isn’t something novel and other platforms benefited from it immensely. In Ruby we have pretty mature Rack which allows hosting Rails, Sinatra or any other rack compatible application on a web server of your choice. In Node.js world there is Connect middleware with frameworks like Express and Zappa built on top of it. To be fair Connect is not exactly the same thing as Rack but the analogy serves well to understand underlying ideas.

Having a common middleware and frameworks built on top of it not only allows you to swap your web server at whim. It also means that you’re now able to combine many application into one easily even if they are implemented in different frameworks. It took me some time to realize how powerful concept it is until I built a demo for a conference presentation. I wanted to have single web server running in background so that I can easily switch between pages built with Angular, Backbone and Knockout. I also didn’t want to throw out separate templates scaffolded with Yeoman and have a shared REST-like api built with Express. How surprised I was when the only thing I needed to do was to call equivalent of:

connect.use(angularApp);
connect.use(knockoutApp);
connect.use(backboneApp);

OWIN opens a new world for .NET web stacks

This all may sound blurry for a .NET newbie or maybe even to experienced developer so I’ll give some examples of what is or will be possible to do in a near future.

IIS as option not requirement – having a way to add a full fledged web interface for a windows service was not an easy thing to do. If only you could you use Asp.Net MVC things would be so much easier. Guess what, when Asp.Net MVC will support OWIN and I’m positive it someday will, this will be a no brainer.
Combine Nancy, FubuMVC, pick a name – maybe you would like to use some existing component and not have to rewrite it to match you stack. Or maybe you work on a really big project and some teams prefer one framework over the other. You are right, this already is possible.
Integration tests – say you would like to execute end to end tests over your web application. Right know you’ll either have to deploy your asp.net mvc app to IIS or IIS Express. This means spinning up separate processes even if you would like to run a single test on you development machine. You may be lucky and be using asp.net web api and be able to use HttpSelfHostServer but then again you’re limited only to this stack. With OWIN it already is possible to use a lightweight Nowin or Katana to run any compatible application inside your tests.
Faster Tests – is hosting a http server inside you tests too expensive? With OWIN you might actually be able to run tests entirely in memory with no network involved whatsoever.
Cross cutting concerns – maybe you need to apply the same authentication mechanism to couple of separate applications. Yes you’re right implementing it as HttpModule might work, but what about this one that you have to support which is based on Nancy. Nancy already supports OWIN so you’re good to go.
Combining Express.js and Asp.Net Web Api – so you’re building a web app using Express.js but you’ve found that it would be nice to reuse some part of the existing code that is built in .NET. With the excellent edge.js and OWIN enabled middleware built on top of it called connect-owin you’re now allowed to this.

One app using both Express.js and ASP.NET Web Api

The last example of OWIN usage seemed like the coolest to me so I wanted to try if it’s really doable at the moment. This wasn’t as easy as one could wish but bare in mind that the OWIN initiative is just starting getting momentum and we’re really on a bleeding edge. The connect-webapi-samples repo contains a sample of single web app where some requests are served by express.js while others using ASP.NET Web Api. You can start the server with:

msbuild ./Connect.WebApi.sln
node ./index.js

Navigating to http://localhost:3000/hello will give you response from Express:

When you go to http://localhost:3000/webapi/values you’ll get a standard ValuesController response:

Note that both requests are logged in console using connect.logger middleware:

I was thinking about a case where this combination could become handy. In a project I’m currently involved in we’re using Highcharts to present beautiful charts to clients. Unfortunately some of them still use ancient browsers like IE7 which does not handle many Highcharts displayed on one page – or rather dies trying to render them. For those browsers we actually produce similar chart images on server. This however requires a lot of boring code to convert from Higchart chart configuration to the 3rd party library format we use for rasterizing.

What we could do very easily with OWIN is to keep a .NET component that generates a complete Highchart configuration and add a very thin “middleware” layer that would take a real configuration pass it into Phantomjs and get back an image that would look exactly the same as real chart. I’m pretty sure it would be much more robust solution than our current approach.

Backbone – when used properly…

Sun, 27 Oct 2013 14:27:00 GMT

...it keeps one’s head out of one’s butt.

This little gem was the first JavaScript MVC like tool that caught my eye. It has a very vibrant community and seems to have reached certain maturity level. More importantly it is so small library that everyone can grok its source code fully in couple of hours. And yet I haven’t actually used it in any project even thought I had couple of occasions.

I guess it was partially because I had read ThoughtWorks technology radar October 2012 edition where author’s claim that:

Backbone.js is a great example of an abstraction pushed too far (…) We find that it blurs the framework and model too much, forcing either bad architectural decisions or elaborate framework hackery in order to preserve sanity.

Back then I had to make a quick decision which framework/library to use, if any, on a project that I started working on and the article steered my away from using Backbone.js. Just for the record the project was developed with Angular.js.

Forming my own opinion

It really bugged me why the technology radar authors stated such a harsh opinion about a tool that has caught some much attention. It seemed even more interesting because I haven’t found a follow up or a more elaborate and concrete comment about the decision to put Backbone.js on hold. I did however find some time to do a simple demo for a presentation I gave on our company internal conference.

The demo I built has only 2 pages. First one shows how to make a page more interactive with a filtered table and couple of elements that trigger the same behaviour. Second view was created just to see how easy it is to make use of html push state. The source code is available on github.

Due to its minimalistic nature it’s really easy to start working with Backbone. The documentation is easy to follow and if you don’t find enough explanation you can always dive into reading the source. The library let’s you choose how you would like to render you views. Out of the box it provides support for client side templates by leveraging underscore.js.

Views are in fact the main force that drives code control. As there is no typical Controller I found it hard to decide where to put some of the initialization logic. For simple cases Router might be a good choice however as the application grows I pretty sure this approach would not be sufficient. It also means that you are now responsible for composition of smaller views into a complete page. I know this problem is solvable in many ways – Derick Bailey’s plugin Marionetteis an excellent addition that fills the gaps not provided by Backbone.

What I find most disturbing is the way we attach events to DOM elements by providing selectors. I use this approach on daily basis in a project solely based on jQuery but there are several shortcomings that in my opinion make it hard to apply to large applications. First of all it means that you typically would add artificial CSS classes to elements. This isn’t bad at first as long as you use those classes only for attaching events and not for providing ways to style your application. An important implication follows –it’s nearly impossible to have a UI designer to work on html and CSS parts without breaking application behaviour. You can however mitigate this by establishing some rules and naming conventions. For instance that all artificial classes used to attach events are prefixed with ‘js-‘ and shall not be used inside CSS files.

Prefer composition over inheritance.

The most important part of the presentation logic are models, view models or presentation models (depending on a pattern flavour you’re using). I think it should be the main place where your application behaviour is defined. Whenever a tool that I’m using is trying to push some structure over me in that single place I immediately get suspicious. Unfortunately it’s not different in Backbone. The way you define and implement your models inside Backbone app is through deriving from base class provided by the library. By doing this you gain many of the commonly required features of model object like filtering, validation and even associations. It speeds up a development a lot especially when you are in need of this commonly requested functionality like form editing in CRUD applications.

However, from my experience, preferring inheritance over composition hurts the most when used to built model part of an client side MVC. I guess this is true for me mainly because I often change the structure and the way different parts of model talk to each other way more often than views and controllers interactions. Having a predefined structure, imposed by framework, limits ways you can define the model and this sometimes leads to pain points that are hard to overcome. In general inheritance is much stronger form of coupling than composition even in a language as malleable as JavaScript so I think that my observations might be true for others too. As all ways this is a question of getting the right balance between having a lot of features ready and waiting for you and keeping you model as flexible as possible to survive the harsh realities of requirement changes.

Don’t listen to me, try it yourself…

I’ve pointed some of, in my opinion, weak points of Backbone. I realize that I might be wrong in some parts and have missed out some important aspects. The fact that the community behind the tool is so active and from what I see always willing to help some newbies can probably out weight cons I’ve listed. I think Backbone is a very effective tool for many programmers out there – it just does fully adhere to my rules.

However when choosing a framework to use on a project one should not be focussed on finding good or bad opinions on the web. I think its much better to spent a week or couple of days with the team actually using the tool to build real features. This can give an immense feedback generated by the project team within actual context – one that you would never get reading reviews on the web.

AngularJs modules to the rescue

Mon, 16 Sep 2013 17:44:00 GMT

Building large scale JavaScript application is a tough problem. The language nature is malleable and it does not have proper modules mechanism built in. Some argue that you can’t really built and maintain big applications built with JavaScript. Others say that the key in succeeding is to never built big applications but to approach things in a more clever way – by applying divide and conquer rule.

Dividing code base

Due to lack of modules mechanism in ECMAScript 5th edition, which is the version most commonly available in browsers, there were couple of implementations created. RequireJs and AMD are just 2 most popular ones. Angular team decided to provide it’s own implementation of modules and left the asynchronous loading of them to existing tools. I suspect that it was because they didn’t want to create a dependency on 3rd party library and because of their use of dependency injection. The framework relies on modules heavily making it possible to use just the parts that you actually need saving precious network and loading time. Here is how to define a module:

var myApp = angular.module('myApp', ['ngResource']);

myApp.factory('itemsStorage', function ($resource) {
  var itemsStorage = ...;
  // items storage implementation
  return itemsStorage;
});

We’ve declared a new module called myApp and stated that it depends on another module called ngResource. In line 3 we are declaring a factory for itemsStorage that requires $resource</em> service. Angular will automatically resolve <em>$ resource service and inject it as a value of parameter. As you may suspect under the hood the dependency injection mechanism inspects parameter names of our function and thus it is able to know how to satisfy them with proper values. Be careful when using JavaScript minimizing tools because most of them will by default shorten parameter names and that will break the DI. The only work around for it is to disable an option responsible for parameters in minimizer. Most of the times I find it useful to split one module definition across couple of files. It is a supported scenario but there are some quirks to it.

var myApp = angular.module('myApp');

myApp.directive('timeAgo', function(){
  return function($scope, $element, $attrs){
    // timeAgo directive link implmentation
  };
});

The first line says that we would like to retrieve myApp module so that we can extend or use it. This API may be confusing, it was for me, as it may look like we’re actually declaring a module that has no dependencies. The distinction between extending and declaring a module has an important ramification. You’re now responsible of loading file with module declaration before files that extend module functionality. In order to declare module with no dependencies you have to pass an empty array as a second argument:

var secondModule = angular.module('secondModule', []);

A praise of dependency injection

Dependency injection is a very useful concept especially in statically typed languages as C# or Java. It really helps you decouple your code as your classes no longer require to know about the implementation details of their dependencies. There are several ways to implement this pattern however must commonly used is a constructor injection. A lot of IoC containers exists both in .NET and Java world: Castle.Windsor, StructureMap, Guice or Ninject are just a few. However I have never seen a viable implementation of DI in JavaScript until AngularJs.

At first it seemed like a magic, so much that I actually had to dig into the framework code base to see how it’s done. As it turned out the idea behind the implementation is really simple. I did however encounter problems while trying to figure out why a particular service cannot be resolved or a directive isn’t working. Usually it was because of typos in parameter names or a missing dependency in module declaration. I know that the Angular team has put effort to make it easier to figure out those kind of problems in a recent release.

I’ve heard opinions that in a dynamic language as JavaScript the DI provides little to know gain instead adding an accidental complexity. I do agree that it is possible to live without however it requires much more discipline to keep a code clean. Whenever you create a service (a piece of code) that will be used in several places you have to make sure not to leak implementation details to clients (the ones that use the service). Without DI it means you have to make it easy to create a service by hand, typically by providing a factory method or a parameter less constructor. If you won’t take care of it upfront it will be harder to refactor the service implementation later on because of the number of clients that create it. When using DI at least this one task is forced upon you upfront.

Now what makes an actual implementation of DI practical is its ease of use. If you remember a time when most of C# and Java IoC containers required declaring actual object graph in an XML files you’ll understand what I mean. I think Angular team made using dependency injection feel easy and very natural and that’s what makes it so useful.

Dependency resolution and conflicts

There is an important limitation in Angular modules implementation around conflict resolution. Basically the framework uses last wins approach so you really have to be careful while naming things. It may seem not important at first but let’s consider following code sample.

var A = angular.module('A', []);

A.factory('serviceA', function(){
 return function(){
  console.log('using serviceA defined in module A');
 };
});

var B = angular.module('B', ['A']);

B.factory('serviceB', function(serviceA){
 return function(){
  console.group('using serviceB defined in module A');
  serviceA();
  console.groupEnd();
 };
});

var C = angular.module('C', ['A']);

C.factory('serviceA', function(){
 return function(){
  console.log('using serviceA defined in module C');
 };
});

C.factory('serviceC', function(serviceA){
 return function(){
  console.group('using serviceC defined in module C');
  serviceA();
  console.groupEnd();
 };
});

As you can see we have 3 modules here A, B and C. But in line 21 inside C module definition we’re declaring serviceA. Since module C depends on A that already defined serviceA we’re actually overriding the implementation. It’s quite useful to be able to override and stuff our custom implementation of particular service provided by other module. Except that we have no control over the scope of our modification. In the above example the serviceA defined in module C will be the one used everywhere at runtime. Even though module B knows nothing bout C. Calling declared services like this: ``` serviceA(); serviceB(); serviceC(); ``` Will produce following output:

You can play around with this sample on jsfiddle.

What can we do about it? The only advice I’ve seen is to use prefix on service names. Now this is a working and simple approach however the code will not look great. I wonder how hard it would be to modify angular injector part to gain the ability to gain more control over overriding things…

AngularJs – my new superhero tool.

Sun, 15 Sep 2013 11:53:00 GMT

As I mentioned in my previous post I’ll describe my thoughts and findings about 3 popular JavaScript libraries that help building large web application. I’ll start with AngularJs which is the first one I’ve used in a real product.

Model View Whatever

Angular team didn’t try hard to map their concepts to classical MVC pattern. You can see it even in the title of its home page. Instead they focused on making a development path of a large web application as smooth and productive, in a long term, as possible. This means its really important to return to documentation and developer guide if you feel you’re getting lost along the way. Learning Angular is a really interesting journey especially that many concepts that I previously had categorized as not for the web turned out to be totally useful.

Model and view model

This term has really a short definition according to angular way:

a model is any data that is reachable as a property of an angular Scope object

The nice thing about it is that it really is up to you how you choose to encapsulate the most valuable and complex logic inside your model or view model classes. No base “class” , no artificial properties and helpers is required. In fact the framework won’t force you to build a proper view model for data-binding. While this freedom gives you more flexibility it also means that you are responsible for keeping your Controllers clean and thin. I prefer to put logic that encapsulates data and provides global, shared behaviour into separate classes totally unaware of the UI. Depending on the use case I’ll then put an instance of this model into scope or if the presentation logic is more complex create a view model that wraps some model objects. To make the model objects easily accessible and sharable between views angular provides services. For a very simple logic that augments the way a single piece of data is presented there are filters.

One important thing to understand though is that data-binding is only happening between view (and directives) and a scope object. If a model object provides events fired when its state changes they have to implemented in a traditional way. When I say traditional I mean that you typically will need to apply Observer pattern – or rather use one of implementations available. This is somewhat different from the dirty checking mechanism that sits behind all the angular data-binding magic. Fortunately the framework provides a way to connect a non angular event to the scope with [ $apply](http://docs.angularjs.org/api/ng.$ rootScope.Scope#$apply) method. However be warned that you have to be really careful here to call it only outside of angular digest cycle.

Controller – keep it thin

A controller is mainly responsible for setting up the scope object and as such is created on a per view basis. It is the most common entry point for an application behaviour triggered by DOM events. While its often easy to put more and more business logic inside them you should really try hard not to. As a reward I often found myself being able to reuse more logic than I anticipated. Another rule of thumb is to never access DOM elements from controller. Angular has a special place for all DOM related activities called directives.

Directive — a way to reusable ui

This is the part of angular that causes lot of confusion when it comes to building custom directives. Especially that whenever you need to manipulate DOM in any way you should use them. To make things easier there is a lot of not so obvious vocabulary used like compile, interpolate, link and my favourite transclude. In my opinion directives make working with data-binding in angular so productive. They are kind of hard to understand deeply at first, but reading the guide couple of times helps a lot. To demonstrate its power I’ll just say that entire jQuery ui is easily accessible with just one directive. I guess I have to little experience to provide some tips about building directives - the most complex one I’ve built was paginate. Here is a HTML snippet of how to use it:

<div paginate="items">
  <ul>
      <li ng-repeat="item in pager.paged">
          {{ item.index }} {{ item.text }} 
      </li>
  </ul>
  <button ng-click="pager.previous()">Previous</button>
  <span ng-repeat="page in pager.pages"
           ng-class="pager.classForPage(page)">
    <a ng-click="pager.goTo(page)">{{ page }} </a>
  </span>
  <button ng-click="pager.next()">Next</button>

Directive is the hardest part to fully understand but if you do learn them and built them in a generic way you can accelerate development a lot.

Views

I’ve only touched couple of aspects of Angular - there is a lot more to it. In particular I haven’t commented on View part in MVC trio. Angular tends to be criticised for the way it handles views. Mainly because you may get to a stage where it is hard to see actual html inside tons of directives attached to DOM elements. An alternative approach would be to use existing template engines like Handlebars – that’s the way it’s done in Ember.js – but it’s not really practical in Angular.

However I really like the idea that behaviour of html elements is immediately visible when you scan views mainly because declarative nature of directives. It makes it possible for an CSS or UI design expert, not familiar with Angular nor JavaScript, to make safe changes to views. In addition you no longer need to repeat selectors to access DOM and attach behaviour to it. This eliminates some of the maintenance burden. Like with any tool you have to be careful though not to overload html with directives and never put logic inside your views even if its only presentation related. If you find yourself repeating the same set of directives in many places of your codebase it may be good to step back a little and think of a way to encapsulate a common behaviour – probably with a more specific directives.

I’ll soon try to comment on features that Angular provides that make it suitable for large scale application development.

Javascript MV* frameworks

Fri, 16 Aug 2013 07:27:00 GMT

Building a web application today isn’t nearly as hard as it was couple of years ago. Back then creating a rich client application using HTML and JS was possible but the number of options you had is a fraction of what is available today. The first SPAI helped building was relatively simple project –built with ExtJs library when it was still in version 2. There were of course other options like dojo but with our team’s little experience in web technology ExtJs seemed like a better choice solely because of its superior support. Shortly after I worked on another web application built with Web Forms and jQuery. We finished both projects, they worked on most browsers but we didn’t avoid creating a mess in JavaScript. Presentation logic was mixed with business rules scattered through event handlers inside unstructured components source code. Just to be fair it definitely was not caused by the frameworks we used but because of our lack of knowledge and experience. I guess that happened to lots of developers and wise people soon enough realized that there has to be a better way of doing things. I think that this was the main reason of the explosion of JS libraries/frameworks options that aim to help structuring client side code.

MVC, MVP, MVVM or …

The number of libraries and frameworks available is overwhelming. AngularJS, Backbone.js, Ember, JavaScriptMVC, Knockout or Sammy.js are just a few I had a chance to look at from the outstanding list. You can even compare how to built simple TODO application with them. Addy Osmani wrote an excellent article that really helps making your choice sane. Most if not all of those frameworks utilize one or more of MV* patterns. An important thing to remember though is that Model View Controller is more an architectural pattern for presentation than an code design pattern like Builder. It means it does not provide very concrete implementation template but rather a guidance for objects/modules/roles relationships. Similar observations apply to MVC cousins namely Model View ViewModel or Presentation Model and Model View Presenter. This can cause some confusion for someone already familiar with MVC pattern in Asp.net mvc, Fubu or Rails. I always try to find a framework/library author’s explanation of how they applied above patterns – it really helps understanding core concepts of their work.

Library or Framework

Some of the tools are called frameworks while other libraries. But what’s the difference anyway? A very common and brief explanation is the one provided by Martin Fowler inside Inversion of Control article:

Inversion of Control is a key part of what makes a framework different to a library. A library is essentially a set of functions that you can call, these days usually organized into classes. Each call does some work and returns control to the client. A framework embodies some abstract design, with more behavior built in. In order to use it you need to insert your behavior into various places in the framework either by subclassing or by plugging in your own classes. The framework's code then calls your codeat these points.There are various ways you can plug your code in to be called.

Of course the distinction is not always firm and clear. Still, it is important to keep in mind while choosing the righttool for a job. A library will usually give you more control inside infrastructure code but it often requires more work to unify boilerplate gluing code and requires some degree of experience. A framework on the other hand will typically provide you with a setup where you can hook your code hopefully focused mostly on the valuable functional stuff. A framework will limit your freedom to some extent in turn it will eliminate some part of work that you presumably would have to do in order to keep your design clean.

Productivity

For me the biggest advantage of using a modern MV* JavaScript framework is productivity. After some time I realized that the amount of boring, repetitive code focused only on filling DOM with data and giving it behaviour is substantial. Sure I am able to separate presentation concerns rather cleanly without aid from any tool. However what’s the point if so many great libraries and frameworks, created by way more experienced people, is out there. It gets even more valid point when working on a big project. Even if all the team members are experienced and closely collaborating it will be hard to have and maintain a common, well understood structure among different parts of JavaScript code base just because understanding and preferences about MVC and related patterns will vary. In my next blog posts I’ll try to review 3 tools that caught my eye. Namely AngularJS, Backbone.js and Knockout that seem to be most popular these days. They all show some signs of maturity and have vibrant community around them. It will by no means be a comparison as they really are different beasts built with different goals in mind. The only comparison I may be tempted to do is about productivity. Still I already know it will not be comprehensive as there will be only one person working on the codebase and the micro product will be focused mostly on a rich data visualization – which is what I do mostly now. Maybe it will help to the framework/library that we’ll use in our current project.

URL Rewrite problem with POST request

Wed, 31 Jul 2013 18:15:00 GMT

Greetings

Hello Everyone! In this blog I would like to share my thoughts and findings on programming, software and a like. I will post here so that I can improve my writing skills and find stuff I was working on previously easier.

URL Rewrite

IIS users had waited a bit for Apache equivalent of mod_rewrite. Finally we've got a decent module from Microsoft - Url Rewrite. Among basic features as rewriting all URLs to extension-less equivalents, blocking unwanted URLs or even more common providing friendly URLs the module provides a nice GUI inside IIS Manager so you can quickly test you ideas without referring to documentation. More importantly rewrite rules of your web site can be configured through Web.config configuration section.

The feature I like the most is the ability to create a reversed proxy inside an IIS web site with a little help of another IIS module - Application Request Routing. I have used it couple of times, mostly to be able to make "cross domain" POSTs. One important thing is to remember to enable proxy in ARR settings inside IIS Manager. Following command line will do it for you:

%windir%\system32\inetsrv\appcmd.exe set config -section:system.webServer/proxy /enabled:"True" /commit:apphost

Interfering with ASP.NET modules

URL Rewrite is a native IIS module and from my understanding it deals with HTTP on a pretty low level. Most of custom IIS modules that I've used or built were written in managed code. This means that they use ASP.NET stack that provides some higher level API to deal with requests and responses. However some of those APIs have important side effects one can easily overlook. Why is that important? Because combining several modules into one request/response pipeline can cause hard to debug bugs.
I've came a cross the same problem twice - as it turned out - it was caused exactly by this interference. Specifically after some happy months of rewriting http POSTs using URL Rewrite suddenly features using it stopped working. In both cases the problem was caused by seemingly not intrusive call:

var request = HttpContext.Current.Request;
var parameter = request["formOrQueryOrServerVariableKey"];

As you may already know this code looks for a value inside QueryString, Form, Cookies, or ServerVariables collection member specified in the key parameter. The important part is a Form which is an abstraction over html form that typically translates to http POST body. ASP.NET reads the body of a http request and decodes form url encoded values into NameValueCollection. This is done by reading input stream of the request which internally is buffered on disk. However since this actually is a network stream it can only be read once. This of course means that depending on the order of modules and at which stage of request pipeline one module decides to read it the other ones may not be able to access it. Exactly this scenario caused URL Rewrited POST request to hang until timeout.

Glimpse

One of the modules affected by this problem is part of a great tool called Glimpse. After an hour of looking through code base of this diagnostic tool I've found what I was looking for inside RequestMetadata class. Specifically RequestIsAjax method that in turn is called by AjaxPolicy during BeginRequestevent. I've created a pull request that hopefully will fix the issue.

One thing that I definitely will remember from this investigation is to always be very careful while writing ASP.NET http module.

Piotr Mionskowski

First steps with AWS Bedrock

Request Bedrock model access

Define your AWS Lambda function with aws-cdk

Invoking Bedrock amazon.titan-text-express-v1

Titan Text Express configuration

Summary

Migrate EC2-Classic RDS to a VPC - Step 2 - DNS

The RDS endpoint DNS

Private hosted zone

Custom DNS record for database endpoint

Important considerations

Migrate EC2-Classic RDS to a VPC - Step 1 - Replication

The source EC2-Classic RDS database parameters

Create a replication user with AWS-CDK

Setup replication slots at source RDS database

Create snapshot at the original database

Create a new database from snapshot

Finding out where to start replication from

Summary

Migrate EC2-Classic RDS to a VPC - Planning

The plan

Step 1: Replicate database

Step 2: Setup a custom database host DNS record

Step 3: Migration!

Conclusion

5 Top Programming Books For Junior and Mid-level Developers

Autoscaling AWS EKS cluster with custom Gitlab runner configuration

Prerequisite

Create EKS Cluster with aws-cdk

Install Gitlab runner using Helm chart

Install Cluster autoscaler using aws-cdk construct

Connect to the cluster with kubectl

Deploy aws-cdk app to different environments

Using multiple AWS accounts to deploy copies of the same aws-cdk application

Deploy aws-cdk app using explicitly specified profile

Benefits of using multiple AWS accounts to deploy aws-cdk app

Downsides of using multiple AWS accounts to deploy aws-cdk app

Using single AWS account to deploy copies of the same aws-cdk application

Using DEPLOY_ENV variable inside aws-cdk entry point

Using DEPLOY_ENV consistently inside aws-cdk application

Combine AWS Step Functions with CloudWatch Events using aws-cdk

Define step function

Define workflow in AWS Step Function

Call Step Function callback from AWS Lambda

HTTPs on CloudFront using Certificate Manager and aws-cdk

Get a trusted certificate from AWS Certificate Manager using aws-cdk

Attach a certificate to CloudFront distribution using aws-cdk

Deploy fast, static website to AWS like a pro

Have a static website

Add aws-cdk to the project

Deploy static content deployment to S3 website

Improve performance of static website with CloudFront

How to convert an express app to AWS Lambda?

An express app

Convert the express app to AWS Lambda

Deploy the express app to AWS Lambda

Add API Gateway to call AWS Lambda

Test API Gateway calling AWS Lambda

Reduced infrastructure cost

TypeScript tips from Kotlin developer - const

Use const whenever possible

Compilers love const

Compilers catch bugs with const

How to deploy Lambda function with CloudFormation?

Step 1: Define a template

Step 2: Create the AWS Lambda function

Step 3: Update and version AWS Lambda function

How to call a load balanced ECS service?

Deploy an ECS service to multiple hosts

A private load balancer

Attach ECS service to an Application Load Balancer

A private DNS record

How to deploy a service to Amazon Elastic Container Service with CloudFormation?

ECS Cluster definition

ECS Service and Task definition

In-memory database tests with Querydsl

Querydsl is awesome

Abstract the complex away

Define production implementation

Use `const` whenever possible

Compilers love `const`

Compilers catch bugs with `const`