Coder Thoughts on software, technology and programming.

Piotr Mionskowski

  • The importance of timeouts

    23 October 2017

    Timeouts are not an exciting thing to talk about. They do not add immediately perceivable value. They are difficult to guess get right and force one to consider problems that are hard to solve. In fact, in my experience, the timeout is only ever considered when our software stops working or is about to. That is an enormous shame since, in my opinion, carefully applied timeouts can vastly improve software resiliency.

    Man with a wrist clock

    An example application

    Let us consider a simplistic example of a Spring Boot application generated using Spring Initializr. The application will only expose actuator API which by default define a health check endpoint. Our example will also have a mail module configured.

    The dependencies section of build.gradle:

    dependencies {
    	compile('org.springframework.boot:spring-boot-starter-actuator')
    	compile('org.springframework.boot:spring-boot-starter-mail')
    	compile('org.springframework.boot:spring-boot-starter-web')
    	compile("org.jetbrains.kotlin:kotlin-stdlib-jre8:${kotlinVersion}")
    	compile("org.jetbrains.kotlin:kotlin-reflect:${kotlinVersion}")
    }
    

    A typical health check endpoint verifies that application integration points work correctly. If the service talks to database then a connection is established and verified. A free disk space is checked. If the service sends emails through SMTP then a connection to a server is established.

    The health checks are auto discovered and enabled when you include Spring Boot Actuator. By default, /health path is used for the endpoint. The SMTP server host, port and credentials obviously need to be configured. At minimum host entry is required e.g. spring.mail.host=localhost. For the debugging purpose one can disable actuator security with management.security.enabled=false to verify what health checks are performed by actuator.

    A last part of our example is the trivial application code:

    @SpringBootApplication
    class Application {
        companion object {
            @JvmStatic
            fun main(args: Array<String>) {
                SpringApplication.run(Application::class.java, *args)
            }
        }
    }
    

    When you request /health the API will return response similar to:

    HTTP/1.1 200 
    Content-Type: application/vnd.spring-boot.actuator.v1+json;charset=UTF-8
    Date: Mon, 23 Oct 2017 08:08:32 GMT
    Transfer-Encoding: chunked
    X-Application-Context: application
    
    {
        "diskSpace": {
            "free": 105755779072,
            "status": "UP",
            "threshold": 10485760,
            "total": 499046809600
        },
        "mail": {
            "location": "localhost:-1",
            "status": "UP"
        },
        "status": "UP"
    }
    

    An application health API, like the one in our example, is often hooked into external monitoring software. The monitor asks the target application about its health in regular intervals e.g. every 5 seconds.

    Shooting yourself in the foot

    The above example has an issue that can kill production server. More importantly, other metrics that are usually monitored e.g. CPU and memory usage will not warn about upcoming, dreadful service stall. The application will also not suffer from an enormous number of requests or emails being sent.

    Imagine that the health endpoint is checked every 5 seconds and that there is intermittent issue with the SMPT server. The health endpoint will rightfully try to connect to the SMTP server and from time to time respond with error. From my experience, when a health check is introduced it typically takes a while to tweak the monitor thresholds so that we get rid of false alarms. It is thus very easy to ignore intermittent errors when we think they are caused by too sensitive thresholds. However, the ignored errors can after a while cause our server to stop responding to any request.

    Man with a wrist clock

    Why this can happen you ask and I answer. There is no timeout configured!

    The mail server health check uses javax.mail.Service.connect under the hood. For a variety of reasons an attempt to establish a TCP connection can take arbitrary longer than usual. Unfortunately the default timeouts used by the javax.mail.* are infinite. A thread that waits for the connection to be established cannot serve other requests even though it barely uses any CPU. The default maximum thread pool size used by embedded Tomcat in Spring Boot application is 200. Assuming that the blocked connection attempt happens twice an hour our application will stop working after 4 days.

    Never use infinite timeouts

    As you can see it is very easy to miss a need for a timeout to be configured. To be fair the Spring Boot documentation states clearly:

    In particular, certain default timeout values are infinite and you may want to change that to avoid having a thread blocked by an unresponsive mail server:

    spring.mail.properties.mail.smtp.connectiontimeout=5000
    spring.mail.properties.mail.smtp.timeout=3000
    spring.mail.properties.mail.smtp.writetimeout=5000
    

    In my option any library or framework should either force the programmer to configure the timeout or have some sensible default. Unfortunately this is not always possible to introduce them later on without breaking changes hence we should check what are the timeouts used when calling any external service.

    Timeouts needed everywhere

    Imagine a controller action method that inserts a single row into a database. Let us further assume that the endpoint is called 50 times per second and it typically takes 100ms to complete. Things work well until we encounter an intermittent sloppiness of the database and now the insert takes 2 seconds to complete. The clients calling the API do not slow down. More request threads are blocked and more database connections are taken out of the pool. Soon as all database connections are in use and all other API endpoints start to fail. This is an example of cascading failure i.e. a problem in one component propagating to others. It is easier to avoid such issues when there is a timeout configured on the controller action and the interaction with the database.

    Every API endpoint should have a timeout configured if we want our services to be resilient. Unfortunately this rule is easy to neglect. Moreover, some frameworks do not even expose such ability. Even in the immensely popular Spring Mvc I could not find a way to set such timeout other than using an async method. Fortunately there are libraries e.g. Hystrix that tackle that problem and can be integrated easily.

    Just to recap here is a short and incomplete list of cases where a timeout should be configured:

    • a controller action
    • a database query, statement
    • a database pool interaction
    • a thread pool interaction
    • an API client e.g. HTTP, SOAP, SMTP

    I will describe how to deal with the cases in the following posts.

  • How to clear database in Spring Boot tests?

    13 October 2017

    Nowadays using a production like database in unit1 tests is a common practice. Calling a real database can increase our confidence that a tested code actually works. Having said that a database, by its very nature, brings external state into a test that will affect its behavior, hence we need to pay special attention to prepare the test execution. There are couple of ways to handle the database state in tests and I’m going to describe an approach I like most.

    Database

    Problems with Spring Boot Transactional tests

    Spring Boot offers many helpers to make testing application easier. Among many you can use a @DataJpaTest which by default will configure an in-memory embedded database. You can use a production type database in tests by adding @AutoConfigureTestDatabase(replace=Replace.NONE) like so:

    @RunWith(SpringRunner.class)
    @DataJpaTest
    @AutoConfigureTestDatabase(replace=Replace.NONE)
    public class ExampleRepositoryTests {
    
        // ...
    
    }
    
    

    The @DataJpaTest uses @Transactional under the hood. A test is wrapped inside a transaction that is rolled back at the end. This means that when using e.g. Hibernate one needs to pay special attention to how the tested code is written. As shown in the Java example below, a manual flush is indeed required:

    @RunWith(SpringRunner.class)
    @ContextConfiguration(classes = TestConfig.class)
    @Transactional
    public class HibernateUserRepositoryTests {
        ...
        @Test
        public void createUser() {
            // track initial state in test database:
            final int count = countRowsInTable("user");
    
            User user = new User(...);
            repository.save(user);
    
            // Manual flush is required to avoid false positive in test
            sessionFactory.getCurrentSession().flush();
            assertNumUsers(count + 1);
        }
    }
    

    Using @Transactional annotation on tests is certainly easy but I still don’t use it for the following reasons:

    • The production code is using transactions with different scope.
    • It is easy to forget about a flush and thus have false positive in test.
    • On failure and when debugging it is hard to see what values were actually saved in db.
    • It is much harder to write tests of production code that requires a transaction to be committed.
    • The test code needs to be more tightly coupled to production code and we all know that it hinders refactoring.

    Cleaning database with SQL

    In tests involving a database I reset its state before each test using plain old SQL. This makes the test code less dependent on how a transaction is scoped inside production code. Furthermore, one can easily review the values saved after a test failure. It turns out it is easy to write a JUnit @Rule or BeforeEachCallback that will remove all rows from all tables. Moreover, we can do so without hard coding table names which would increase maintenance cost.

    Let’s start with defining a @Rule in Kotlin in that will be called before each test:

    import org.junit.rules.ExternalResource
    import org.springframework.stereotype.Component
    import javax.sql.DataSource
    
    @Component
    class DatabaseCleanerRule(private val dataSource: DataSource) : ExternalResource() {
    
        override fun before() {
            if (databaseCleaner == null) {
                // Consider inspecting dataSource to check if we are connecting to test database
                databaseCleaner = DatabaseCleaner(dataSource::getConnection)
            }
            databaseCleaner!!.reset()
        }
    
        companion object {
            internal var databaseCleaner: DatabaseCleaner? = null
        }
    }
    

    Consider inspecting dataSource to check if we are about to connect to test database and not one used for development. It is very easy to use incorrect Spring Profile and wipe out your development data. Ask me how I know?

    We can use the DatabaseCleanerRule in a spring enabled test as any other JUnit rule e.g. @Rule @Inject lateinit var cleanerRule: DatabaseCleanerRule.

    Notice that we’ve delegated the actual important work to DatabaseCleaner class defined in Kotlin below.

    import com.practi.util.iterator
    import org.slf4j.LoggerFactory
    import java.sql.Connection
    import java.sql.PreparedStatement
    import java.sql.SQLException
    
    class DatabaseCleaner(private val connectionProvider: () -> Connection) {
        private val tablesToExclude = mutableSetOf<String>()
        private var tablesForClearing: List<TableRef>? = null
    
        fun excludeTables(vararg tableNames: String) {
            tablesToExclude += tableNames.flatMap { listOf(it, it.toLowerCase()) }
        }
    
        fun reset() {
            if (notPrepared) {
                prepare()
            }
            executeReset()
        }
    
        private val notPrepared get() = tablesForClearing == null
    
        private fun prepare() {
            connectionProvider().use { connection ->
                val metaData = connection.metaData
                val tableRefs = metaData.getTables(connection.catalog, null, null, arrayOf("TABLE")).use { tables ->
                    iterator(tables::next) { tables.getString("TABLE_NAME") }
                        .asSequence()
                        .filterNot(tablesToExclude::contains)
                        .map(::TableRef)
                        .toList()
                }
    
                tablesForClearing = tableRefs
    
                LOG.info("Prepared clean db command: {}", tablesForClearing)
            }
        }
    
        private fun executeReset() {
            try {
                connectionProvider().use { connection ->
                    val reset = buildClearStatement(connection)
                    val result = reset.executeBatch()
                    result
                }
            } catch (e: SQLException) {
                val status = engineInnoDbStatus()
                LOG.error("Failed to remove rows because {}. InnoDb status: {}", e, status)
                throw e
            }
        }
    
        private fun engineInnoDbStatus(): String { ... }
    
        private fun buildClearStatement(connection: Connection): PreparedStatement {
            val reset = connection.prepareStatement("")
            reset.addBatch("SET FOREIGN_KEY_CHECKS = 0")
            tablesForClearing?.forEach { ref ->
                reset.addBatch("DELETE FROM `${ref.name}`")
            }
            reset.addBatch("SET FOREIGN_KEY_CHECKS = 1")
            return reset
        }
    
        data class TableRef(val name: String)
    
        companion object {
            private val LOG = LoggerFactory.getLogger(DatabaseCleaner::class.java)!!
        }
    }
    

    Notice that we’ve defined tablesToExclude set that allows us to omit certain tables. This comes in handy when you’re using a database migration tool that stores its state inside some table(s).

    The JDBC metadata allows us to introspect schema regardless of the database vendor. The iterator is a tiny Kotlin function that aids consuming iterator like objects:

    inline fun <T> iterator(crossinline next: () -> Boolean, crossinline value: () -> T): AbstractIterator<out T> = object : AbstractIterator<T>() {
        override fun computeNext() {
            if (next()) {
                setNext(value())
            } else {
                done()
            }
        }
    }
    

    The buildClearStatement constructs a large query that DELETEs all rows from each relevant table. The example above uses MySQL where it is very easy to disable foreign key checks. This is important since foreign keys would prevent rows to be removed unless we paid special attention to the order of removal. A more generic example of how to deal with referential integrity when clearing a database can be found in the Respawn project.

    Last but not least, when a SQLException is thrown we log the exception accompanied with SHOW ENGINE INNODB STATUS. The status information can hint us about failure reason e.g. another test process executing against the same database or a rogue, runaway thread that locks some rows.

    private fun engineInnoDbStatus(): String {
        return connectionProvider().use { connection ->
            connection.createStatement().executeQuery("SHOW ENGINE INNODB STATUS ").use {
                iterator(it::next) { it.getString("Status") }.asSequence().toList()
            }.joinToString(System.lineSeparator())
        }
    }
    

    The above examples show that it is not hard to manually reset the database. I’ve found that using this approach makes my tests more trustworthy and less coupled to the underlying persistence layer. In fact, we can easily switch e.g. from JPA to JdbcTemplate in a performance critical code area without a need to change a test.

    1 Whether it is actually unit or integration test is a different topic.

  • Simple Config Sections 1.2 released

    25 December 2015 .net

    About a week ago I’ve released another version of SimpleConfigSections. It’s marked 1.2.0 and you can install it through Install-Package SimpleConfigSections.

    What is SimpleConfigSections?

    SimpleConfigSections is a little library that simplifies defining .net configuration sections. As you can see in the mentioned msdn documentation the amount of boilerplate code is substantial. An equivalent example in SimpleConfigSections looks like follows:

    As you can see the SimpleConfigSections version is more concise and a bit easier to write. More examples can be found on project’s github page.

  • Android Data Binding Part 2: Observer pattern to the bone

    27 July 2015 android

    In the previous part I’ve described typical problems we have to face when developing applications on Android. I’ve also highlighted that some of them may be mitigated when data binding API is utilized properly. It’s time to dive into more details of how this promising API works.

    Observer Pattern

    At the root of many of solutions we find in today’s APIs is a design pattern. In case of the discussed Android API it is Observer Pattern applied to the bone. In fact this particular pattern is so common and powerful that some languages and runtimes (C# with .NET, Objective-C and Swift on iOS and Mac) provide neat support for it. You may wonder why it is important in the context of Android data binding? The answer is simple yet easy to ignore - memory management. The following diagram depicts dependencies of an observer pattern’s elements in the context of Android data binding: Android data binding diagram On the right side we have the observable part producing notifications about changes - an object implementing android.databinding.Observable interface. In the observer pattern events producer can have many events listeners - OnPropertyChangedCallback implementations. Android data binding provides implementations of events listeners taking care of property values changes as well as collection elements changes.

    Note that the Observable object knows nothing about concrete OnPropertyChangedCallback implementations (Dependency Inversion Principle at its best). However, at runtime the Observable object will hold references to many OnPropertyChangedCallbacks instances. If you look further down the diagram you’ll see that in order for OnPropertyChangedCallback implementations provided by Android data binding library to update the state of the view component a reference to it is required. This means that although the view model knows nothing about the view components at compile time they will reference them at runtime.

    How Android data binding library aids memory management?

    As stated above in a simple implementation we would have a lightweight and testable view model retain expensive view widgets - thus making it heavy. In the Android context it means that even though an Activity got destroyed we could have other object still retaining its Views, trying to update them and preventing them from being garbage collected. Not so nice, huh?

    If you look closer at how Android data binding is implemented you’ll immediately see that its OnPropertyChangedCallback holds only weak references to views. WeakPropertyListener, WeakListListener, WeakMapListener and ViewDataBinding.WeakListener make sure that the Observable object is not retaining views. This means that there’s no need for a developer to manually stop data binding in an activity onDestroy or a fragment onDestroyView methods. If you use RxJava you probably know that this extra step is tedious and requires a great deal of attention and boilerplate code. You can find out more about the problem in at least couple of issues on GitHub: 1, 2.

    Because Android data binding library takes extra steps to make sure a view model will not cause views to be retained we can detach a view model’s lifecycle from that of an activity or a fragment. We now have an easy way to i.e. do a simple in-memory caching via singleton view models. More importantly, we can cross out at least one place when looking for a memory leak cause.

    This article is cross-posted with my company blog

  • Android Data Binding Part 1: Why it is important

    20 July 2015 android

    Android application code often suffers from being more verbose than it could be. As libraries such as Android Annotations and ButterKnife have shown that’s only partially due to tediousness of Java. The recently announced Android Data Binding library can remove at least part of the boilerplate code we need to write. Since I’ve always liked Presentation Model pattern (MVVM) this is very dear to my heart. However just getting rid of a tedious code is not the main reason I’m so happy to see the new API. Let’s recap on common issues developer faces on Android and then I’ll show how using mentioned patterns with new offering from Google can mitigate them.

    Problems with classic approach

    I’ll explain that step by step using a registration form screen example. We need couple of fields like first name, last name and an email address. To make the example a bit more interesting and to make UX better we’ll use floating label pattern. One of the simplest approaches without data binding (for first name only) might look like this:

    Memory leaks

    A seasoned Android developer will immediately spot that we’ve a potentially leaked activity here. The culprit is of course the anonymous inner class implementing RegisterApi callback (Action2), which in turn uses findViewById. However it’s not the mentioned method that retains the activity - in Java every non-static inner class will have implicit reference to the enclosing class instance. While this problem is so common I still haven’t found a concise solution in the Android SDK. There are however community driven libraries i.e. Otto and RxJava that can help tackle this problem.

    To much code in Activity

    Many of us, me including, are guilt of stuffing too much into Activity classes. Writing test and maintaining them can become a nightmare over time. The verbosity of Java only makes the matter worse. One way to slim activities down is to use Presenter pattern written either by hand or with the help of Mortar. Another solution is to encapsulate more logic into custom view classes used in layouts and using shared static helper methods with common code that calls Android API. Nevertheless, for a programmer that just starts with Android, it’s important that the framework provides guidance and samples that encourage separation of concerns - in my opinion up until now that was rarely the case.

    Maintaining layout files

    Many if not most screens in Android (and other platforms, too) are read-only. That is they are not forms and fields that user can change. Instead they offer neatly presented content with some way of interacting with it (as opposed to changing it directly). When using Android layout files we’re forced to use references (auto generated in R class by aapt) that allow setting view’s properties. This isn’t something necessarily bad especially if you need to heavily interact with a View. Having said that, how often you used findViewById (or @InjectView) just to set the content of TextView or had to refactor activity fields type because of a change in a layout definition?

    Android Data Binding can help you.

    The following code illustrates how registration form might look like with data binding:

    The new RegisterActivity is similar to previous example. We inflate the layout differently to initialize data binding and get an instance of ActivityRegisterBinding class. The class, with the name derived from layout name, is generated by data binding and its main purpose is to provide a glue between a view and an observable view model. When using registerApi.register we only reference form variable thus avoid unnecessarily referencing the activity. The RegisterForm class is a view model for our layout and has all the data that activity_register layout requires.

    The view model has 3 observable attributes marked with @Bindable annotation. We don’t strictly need a backing field as the getFirstNameLabelVisibility shows. What we do need is to fire change notification events, which is done via notifyPropertyChanged and notifyChange methods available on BaseObservable. The former method accepts a field id that is assigned by data binding processor in generated class BR (similarily to R class). Deriving from BaseObservable is not a must - to make a model bindable it has to implement android.databinding.Observable. Last but not least a layout file:

    The data binding library, while still in beta, provides a nice toolset to remove the boilerplate code and improve Android code (unit) testability by extracting the logic from activity (or fragment). When used properly it can also save you from leaking resources. In the next post I’ll explain in more detail how it happens.

    This article is cross-posted with my company blog