Skip to main content

Scala - Execution Order when Mapping Over Twitter Futures

As Twitter Futures don't have the concept of ExecutionContext (unlike plain Scala Futures), it is at first hand hard to know what gets executed when and where...

One creates a Twitter Future by specifying the FuturePool in which it will run:
In this case, the content of the code-block will be scheduled to run immediately as the unbounded pool has no limit to the number of concurrently running tasks.

One can also create one's own FuturePool. For example, in the code example below, we create a FuturePool limited to two concurrent tasks:
Now, getting to the crux of the matter, the question is: in what pool does the code passed into the map function run?
Looking at the output copied at the end of the snippet, you will notice that it runs immediately after its corresponding Future has completed. The total number of active tasks stays limited to two, from which we can deduce that it gets run in the same thread pool as its Future.

Could one run the map statements in another pool? Yes, by creating another Future and using flatMap:
Notice that we now have up to four concurrent tasks running as the number of Futures executed inside flatMap are not bound anymore.

Which leads us to the last question: is there a difference between using map and flatMap when we decide to keep using the same pool?

See for yourself:
Do you notice anything different in the output of the TwoThreadPoolAndMap.scala code snippet compared to this one?

The execution order of tasks has changed! Instead of running the statements passed to the map function immediately, first the initial Futures are executed, and then the Futures inside the flatMap function start to run as they are dequeued from the pool. Notice that in both cases the maximum number of concurrently running tasks stays at two.

Hopefully, these gotchas helps you if someday you need to order the execution of Twitter Futures. I have a preference for using flatMap by default as it will probably lead to fewer surprises. Have a look at this answer to a Stackoverflow question by one of the authors of the Twitter Futures if you want to understand more about their threading model compared to Scala Futures.

Comments

Popular posts from this blog

In Defence of the Software Engineer

It is often argued that software engineers are not real engineers as they lack the credentials and formal methods required to claim the title. I will counter that, first of all, these statements are simply false; and second, that this argument relies on a mistaken notion of what it means to be an engineer.

The credentials part is the easiest to argue as I am myself a Qualified Engineer certified by the engineering certification authority of my country, the Commission des Titres d'Ingénieur. Consider me the living proof that it can be done!

Now, that we don't rely sufficiently on formal methods as a profession can be argued either way. Some software developers will reject the idea of being engineers and prefer to call themselves craftsmen. It is true that in many cases a software developer has to come with unique and creative solutions to a problem, making it hard to apply formal methods. On the other hand, a lot of software engineering is rigorous and has a wide overlap with c…

Scala Coding Exercise - A Recursive Implementation of an Immutable List

I recently got to code a list as an interview coding exercise from one of the FAANGs. I thought that it might be a good idea to post it there for future reference.

I went for a recursive and immutable implementation. It's quite naive and I would certainly not use it in production, but as an exercise it was fun writing it due to its elegance.

If you look at the list implementation in Scala 2.12, they went for an imperative solution for most methods... Elegance sometimes comes at too high a price in terms of performance!


So... Do I pass the Fizz Buzz test?