Transcript

The topic of this talk is Incr_dom. This is a framework for writing web UIs that we designed at Jane Street a few years ago and it’s what we’ve been using to write a lot of our internal web UIs. The first half of this talk, I’m going to go over some of the background of the framework as well as some of the main concepts behind it and then in the second half, we’ll do a demo to see how to actually write a simple Incr_dom app. Let’s start with some background.

As many of you know Jane Street is a trading firm. A lot of our coding effort goes into writing trading related software. As a result, we don’t have as many resources or as much time to focus on web development. At the same time, we very frequently find ourselves with large amounts of data that we want to display to the user in a nice way, and that we want users to be able to interact with and modify, so it turns out we want to write web UIs quite frequently. Here’s an example of a typical Jane Street UI. This is an example that’s actually available in the Incr_dom examples directory so you can all access it after the talk if you’re curious. This is a mock up of a trading system GUI. I wouldn’t worry about any of the trading terminology in here.

Basically, all you should care about is that this is a table with a few columns and lots and lots of rows. There are 10,000 rows to be exact. You’ll notice that the data in this table is changing fairly frequently. This doesn’t seem like a really exciting UI, but there are several things that you can do here. For instance, you have this focus that you can move around and notice that it always stays scrolled into view. You can also edit certain cells. Let’s double the edge of EMIK. Also you can filter on the first column. Here are all the symbols that contain sub string ABC, for instance, or AB. Also you can start on any of the columns. Let’s say I sort on last fill. This is what it looks like.

This is a great illustration of how frequently everything is changing in this app. There’s updates tens of times per second and also you’ll notice that the actual order of the rows changes as the content of the column we’re sorting on changes. There are several pretty tricky things about making this app efficient. First of all, as I mentioned before, we just are dealing with a really large amount of data and very frequent updates. Just that fact makes it really hard to get this to be efficient. Also there’s some more subtle things that causes to be tricky. For instance, when we’re sorting the rows, every time the contents that we’re sorting on changes we have to recompute the order of the rows, and same idea with filtering. Every time the content of the column we’re filtering on changes, again, we have to figure out which columns, which rows rather, we want to keep in which ones we want to throw out.

One technique that we use to make rendering somewhat more efficient is partial rendering. I’m going to dig into the HTML to find our table, and here it is. Can you all see that more or less? Okay. I cleaned over 10,000 rows. However, here we only see 10 or 20. One trick we use is partial rendering, where basically we only actually render the rows that are in view at the moment. If I scroll down slowly you’ll see that new rows get added to the bottom and old rows get removed from the top. This is one trick that is not really part of Incr_dom proper, but that’s a common technique that we use to make our apps more efficient and I’ll talk about some other things that are a part of Incr_dom proper as well.

Hopefully this gives you an idea of the types of UIs that we deal with at Jane Street. We wanted to put together a framework that would make our lives easier in several ways. First of all, we wanted it to be really easy to be able to write these web UIs and for it not to require learning all the ins and outs of web development or having to learn JavaScript, we hoped it would be approximately just as easy as writing an ordinary OCaml app. Also, we wanted to be able to write really easily maintainable apps.

The hope was that once a developer put in some work into adding a UI to an existing tool or system that they had, that after that they could kind of set aside the UI and not really have to worry about it anymore and count on it continuing to work even as various things in the backend change. Lastly, and very importantly, we wanted to be able to optimize our UIs. As you saw in the previous example, we often deal with really large amounts of data, very frequent updates so it’s really important that we can get our apps to still be responsive and still update smoothly.

We looked at a lot of existing web UI frameworks out there, and two of the ones that we got a lot of ideas for Incr_dom are React and Elm. React is a JavaScript library written by Facebook and it’s fairly popular nowadays. That’s made especially for writing interactive UIs. Elm is a language that was designed specifically for writing UIs. It was designed as part of a senior thesis at Harvard, which is pretty cool, and it’s a functional language. It’s quite similar to OCaml in fact. I’m going to talk about the actual ideas that we got from each of these libraries in the next few slides but the takeaway here is that Incr_dom was heavily inspired by a lot of existing ideas.

The first concept behind Incr_dom that I’d like to talk about is js_of_ocaml. This is a compiler from OCaml bytecode into JavaScript. It’s very powerful. It can compile pretty much the entire OCaml language, and it allows you to run pure OCaml code within a web browser, which is pretty cool. You could imagine writing a whole OCaml top level that’s run purely in the browser. This was great for us because it meant we could write our web UIs in OCaml and this has several benefits.

First of all, a lot of our backend is already in OCaml so it was very convenient to be able to write our client code in the same language, because it meant we could share a lot of code between the two, and this is great for maintainability because often if something changes in the backend, if that change is made to shared code, it’ll automatically apply to the client code as well and if not, maybe it’ll break some things, you’ll get some compile errors, but those are still a lot easier to deal with and to fix than trying to maintain your web UIs in a completely separate code base.

Another benefit is that we have been working for years to build up tools and libraries to make it easier for us to program in OCaml. It was really beneficial that we are able to use these libraries and tools when we’re writing our UIs. It’s nice that instead of having to reimplement something in JavaScript that we already had in OCaml, we can instead just compile it into JavaScript using js_of_ocaml.

Lastly, we have a lot of OCaml knowledge at Jane Street. All of our developers are extremely comfortable and familiar with it. This meant that onboarding would be just a lot easier for new users of this framework. In fact, we have a lot of reasons why we really like OCaml and why we’ve chosen to use it within Jane Street. All of those benefits could now apply to our web UI writing as well. Things like type safety and the expressiveness of the language. Right now we know what language we want to write our UIs in, but we still no idea of what the actual code should look like. This is where the Elm architecture comes in. This is a design pattern that you can use when writing UIs and it’s commonly talked about within the Elm community. It was first noticed because it frequently just naturally emerged in a lot of Elm programs.

If you’re familiar with the model view controller pattern, this is pretty much the same idea, except that we have a bit more specific information about what that controller bit should look like. Here’s the actual pattern. We have three main pieces that we need to be concerned with. We have a model which is the state of our app at any given point in time. We have the view, which is a way of displaying our app to the user, and for instance, we can do this using HTML and we have some actions which are the ways in which we can update our model. Every time you have a new model, you call this view function to produce your new view for it and you can imagine in your HTML that you have some event handlers that handle things like users clicking on buttons or typing in text, and so the way those event handlers should handle the events is by scheduling actions. Once you have some action scheduled, you eventually apply them to your model using this apply action function.

We can think of Incr_dom as a first approximation as an implementation of this ELM architecture and it provides several benefits. It’s a very nice pattern. One benefit is this fact that we’ve clearly separated out the logic of our program into distinct parts that deal with different tasks. We have a function that generates our HTML for us. We manage the state of our app completely separately from that, and we have a very clear way to update the state. Another benefit is that this uses the decorative paradigm. The idea here is that you’re able to express your model… sorry, rather your view, as an all at once computation. Instead of having to figure out the series of steps that you need to make to your old view to update it into your new view, you can just specify what you want your new view to look like in the end, and you don’t have to worry about figuring out the differences.

Now that we know that Incr_dom is more or less an implementation of this Elm architecture, we can form an idea of what an app interface should look like. To be clear, this is not the interface of the Incr_dom library. Instead, this is the interface of an app that is managed by Incr_dom. Basically the user has to implement an app with all of these components and then Incr_dom will take care of wiring everything together and managing the state and everything else that needs to be done.

Let’s take a look at the pieces that the user has to provide for their app. First of all, there are two types, a Model.t and an Action.t and these correspond exactly to the two components out of the three that we saw in the Elm architecture. The user can specify any types that they want for their model and for their action. Notice, they don’t get to specify the type for the view because that’s already decided for them. It has to be HTML. Next, the user has to provide two functions, the apply action function and the view function. If you recall from the previous slide, there was also a schedule action function. This is provided by Incr_dom itself so the user doesn’t have to implement this.

The apply action basically takes a model and an action applies the action to the model and returns a new model. Then the view function takes in a model and the schedule action function which allows you to schedule new actions from within your event handlers and it produces an HTML DOM. If you’re not familiar with what a DOM is, it stands for document object model. It’s approximately a mutable tree that represents the state of your page. It’s very closely related to your page’s HTML, and it is very closely linked to the browser. It directly determines what the browser displays, and that means that every time you make a change to the DOM, that results in browser work. Things like recomputing CSS, figuring out the new layout and actually repainting everything.

Now we have an idea of what our app should look like. We have an interface that we have to basically implement. However, there is a problem with this, which is that every single time we have a new model, we have to completely recompute our DOM. As I mentioned, any changes you make to the DOM are extremely expensive because they result in browser work, so this ends up just not being feasible. To fix this, we bring in virtual DOM. Virtual DOM is a technique that you can use to avoid making excessive changes to your DOM so that you reduce the amount of work that’s caused for the browser. This is a very popular idea that appears in a lot of frameworks.

In fact, both Elm and React use it. I think React is the framework that has popularized it, but it existed before React. The idea here is that you store a virtual copy of your DOM. You just store a copy of your DOM in memory as a lightweight object where it’s not at all connected to the browser. If you ever make changes to it, it’s just as cheap as any changes you’d make to an ordinary JavaScript object. Unlike making changes to the DOM, it doesn’t result in any work for the browser and so it’s much cheaper to update it.

Here’s the idea. Let’s say that my DOM right now has three nodes, A, B, and C. We see that on the right of this diagram. Let’s say that we have this virtual copy, this virtual DOM of the current state of our DOM, which is labeled old virtual DOM at the top left. Now, let’s say I want to change my DOM to actually have four nodes, A, B, C, and D and recall, we want to use a declarative paradigm where we just all at once specify what we want our new DOM to look like. I create a completely new virtual DOM from scratch that has the nodes A, B, C, D. Then the first step of the different patch algorithm is diffing. We diff the old virtual DOM against the new virtual DOM and we produce a patch that contains exactly the difference between them.

In this case, the difference is just this additional child node D. Next, we apply only the patch to the HTML DOM, and so this ends up being a much cheaper modification to our actual DOM than if we had completely recreated our DOM from scratch. The benefits of this are first of all it is a more efficient way to update our DOM and second of all, it allows us to use this declarative paradigm of specifying what we want our view to look like all at once, which would not be feasible to do if we were doing that directly to our actual DOM.

Now I wanted to show you the API of our virtual DOM library because we will need this when we’re actually filling in our view function in the demo. There’s two main types, an attribute and a node. To create an attribute you need to specify a name and a value, and to create a node you need to specify a tag, a list of attributes, and a list of children which are themselves nodes, so ends up being a tree more or less and each of these two types also has some utility functions just to make the syntax more concise.

I just wanted to show you that this syntax corresponds very closely to actual HTML. Let’s compare some sample HTML to the corresponding virtual DOM. Note that every time we have an HTML tag, we call this Node.create function and pass that tag in as the first argument just as a string. Then if our tag has any attributes associated with it, we pass those in as the second argument. For instance, you can see that on the third and fourth line for our div tag, and finally we pass in whatever children are associated with that tag as the last argument. You can see this for body, it has two children, and we can make this slightly more concise using the utility functions I mentioned. For instance, there’s a function called body that’s a bit more concise than having to call create and pass in body as a tag.

Now that we have this idea of a virtual DOM, we can use it in our app. Instead of our view function returning a new DOM each time our model changes, it can return a new virtual DOM each time our model changes, and this is nice. Now we’ve solved the problem of avoiding all this extra work in our browser that’s very expensive, but it turns out that even just creating a new virtual DOM node every time our model changes is also expensive enough that the apps we get are just not as efficient as we’d like. To fix this problem, we bring in Incremental.

Incremental is an OCaml library that deals with self-adjusting computations. What that means is that it allows us to build a complex computation that depends on some set of inputs in such a way that when some of our inputs change, we’re able to update our computation in an efficient way. In practice, the way this is done is we build up a graph where all of our inputs are nodes in the graph, and for any given node in our graph, its value is a function of the values of its parents. What this means is that if an input node changes, we know that its descendants are exactly the nodes that need to be recomputed.

Firstly, recompute all of its children, then its children’s children, and so on. As an additional optimization, we also have this idea of cutoff where basically if you recompute the value of a node and find that the new value is equal to the old value we know that we don’t actually need to recompute all of its children. This saves a lot of extra work. The way Incremental is used in Incr_dom is to more efficiently convert our model into our virtual DOM. For instance, if only a small part of our model changes, we can then only recompute the small part of our virtual DOM that corresponds to that instead of having to completely recompute our full virtual DOM.

Now I want to show you some incremental syntax so that we’ll be able to use it in our demo. There is this type alpha t that basically represents an incremental computation whose value is of type alpha. Another way to look at it is type alpha t represents a node in our incremental graph where the value of that node is of type alpha. We could have a type entity which represents the type of a node with an integer value for instance. Now we have two functions that allow us to build up our incremental graph, assuming that we already have our input nodes.

The map function basically allows us to add one single new node to our graph that has a single parent. As arguments to the function, we provide the parent node as well as a function from the parent’s value to the child’s value, and so the map function creates this new child node for us and returns it. Map2 is exactly the same idea, but it creates a node with two parents instead of one. We pass both parents in as arguments to our map2 function and a function to get from the parent’s values to the child value, and it creates and returns this child value for us.

Now I want to show you this let syntax, which is a syntax extension. It doesn’t change the semantics of anything. It’s just syntactic sugar but I’ll be using it and it’s very convenient so I wanted to show it to you. Let’s look at the Incr.map function. If you’ll recall, it takes two arguments. One of them is an incremental node and the other one is a function from that incremental node’s value to a new value. If we look at the let syntax that corresponds to this, we first start off with let percent map, and then we have this a variable that is not incremental, it’s constant and then on the other side of the equal sign we have an a_i variable that is incremental.

The variable on the right side of the equal sign corresponds to the first argument we passed to Incr.map and the non incremental value on the left side of equal sign corresponds to the first argument that our function F takes. It’s a little bit confusing, but once you get used to it, it’s very nice because if you ignore all the percent maps in your code, it looks like an ordinary non incremental program.

This is just a ppx that goes through that [crosstalk 00:23:09].

Exactly, yeah. If we want to do this for map2, notice we still just use the keyword map instead of map2 and we also have this additional keyword and that we use to separate the two incrementals, and we can extend this to three incrementals or four or whatever number. We could write out the equivalent of what would be map3, map4, map5 using let syntax. Let’s look at an example now. Let’s say we start off with three nodes that are int incrementals, and we want to add the four nodes below them to our incremental graph. We want a min node that just takes the minimum of the three. We want a max node that takes a maximum of three and we want two nodes min_view and max_view that create very simple virtual DOM nodes that just display those values.

This is what the code would look like. To create our min node, we have to map over all of the incrementals that it depends on. Because it depends on A, B, and C, those are the incrementals you map on. Once we have their values, we apply some sort of minimum function to them. We have a very similar idea with creating our max node, but we apply some sort of max function instead of a min function and then to create our last two nodes, the min_view and max_view, they each have one dependency so we map on the single node that they each depend on. For min_view we map on the min node and then we create a simple text virtual DOM node that just displays that value as a string.

Now let’s put this into a bit more of an Incr_dom context. Let’s say that we’re looking at our view function, which if you’ll recall takes in an incremental model or rather the last time we looked at it, it was just a constant model, but now let’s imagine we can take in an incremental model. Let’s say that these three int incrementals that we started off within the previous slide are actually obtained by projecting them out of our model. Furthermore, let’s say that once we have our min_view and our max_view nodes, we kind of stick them together into a body node to get a virtual DOM node that we can actually return from our view function.

One thing to note here is that because everything is projected out of our input node, which is our model, if we didn’t have cutoffs, then that would mean everything would get recomputed every single time our model changed because everything is a descendant of our model and this is generally true for all Incr_dom apps. In your view function, all of the nodes are descendants of your model. This is where cutoffs become really important. If you’ll recall, a cutoff is this extra check, if a node’s value hasn’t changed after an input has updated you know that you don’t need to recompute its children.

Here for instance, let’s say we had some fourth variable, D in our model that we don’t end up using in this graph. If that value changed, our model would change. We’d have to recompute our A, B, and C nodes. However, then we’d find that our A, B, and C values haven’t actually changed so we’d avoid having to do the rest of the work of recomputing the rest of the graph. Here’s what the code looks like for this. We’re looking at our view function and I’ve simplified it to take a single argument in which is a model incremental. To create our three incrementals A, B, and C, for each of them we have to map over their single dependency, which is the model and then let’s say we have some functions, get_a, get_b, and get_c to extract our prospective ints out of the model.

Then once we have these we’re back to the situation we had in the previous slide. We would reuse the same code we had there to construct the middle part of our graph. Once we have our min_view and max_view nodes, we would map over both of them and put them into a virtual DOM body node to get our final view node that we would return from our view function. Now, I wanted to mention a very useful library that’s built on top of Incremental called Incr_map, and this allows you to efficiently work with incremental maps and to be clear here by map, I mean, a key value store structured as a tree.

Let’s look at an ordinary map function that I’ve called convert. What this does, if any of you are familiar with OCaml, the actual name of this function is mapi, but I’ve changed it to avoid confusion because there are many meanings of map. This takes in a map as an input as well as a function to basically transform all of its data into new values, and it gives you back a new map with all of the values transformed using that function. If we look at the equivalent Incr_map function, you’ll notice that it has pretty much the same signature except that it takes a map incremental instead of a constant map and returns a map incremental. This does a very similar thing to what we’ve been seeing in our view function.

Basically it avoids doing unnecessary work when our input map changes. We have these functional maps and we have an efficient diffing algorithm over them. Every time our input map changes we’re able to only do work proportional to the size of the diff to update our output map instead of having to reconstruct the whole map. This library is extremely useful in Incr_dom whenever you want to display something that you’d normally use a list to do such as cells in a row or rows in a table.

If we add this idea as incremental to our app interface, which we’ve already kind of done, our view can now take an incremental model as an input and return an incremental virtual DOM as an output. This saves a lot of work and you now no longer need to completely recompute your virtual DOM every time your model changes. Now we have an interface that allows us to actually build reasonably efficient web UIs. Let’s look at a demo.

This is what we’ll start with. It’s a very plain looking app. Basically it just has this one heading with the text, My Incr_dom App. We’ll be building on top of this to make it a bit more interesting. Let’s take a look at the code that was used to create this. There’s three main files that you should be concerned with. The first one is dune, which contains all the instructions that the build system needs to know how to build your app. You need to specify things like the OCaml module that you actually want to compile into an executable, which is main in our case. Also any dependencies that our code has. We depend on the Incr_dom library here.

Also this js_of_ocaml line you need to include for any executables that you want to be compiled into JavaScript as well as any libraries that those executables depend on. Finally, we specify any pre processors that we require. Don’t worry too much about what that means if you’re not familiar with pre processors, but for instance, ppx_jane is required in order to use the let syntax that we saw for incremental. So the next file that we need is just this very ordinary looking HTML file. You’ll notice the body’s left empty, that’s because Incr_dom will be generating the body for us.

Also, it’s important to include the JavaScript file that’s going to be compiled by js_of_ocaml as a script in the head, and we can also include some additional stuff if we want. Here I have a style sheet. Now, I guess we have this CSS here, just very basic styling of our tables. The main thing that contains the bulk of our code is this OCaml file main.ml. The first thing we do in this file is to implement an app module that has the interface that you’ve been seeing on all of the slides. Additionally, we also provide an initial model just to specify what the initial value of our model should be when we start up our app.

Let’s take a look at this. First we have our type Model.t which I’ve set to be equal to unit. This is like void in other languages. It just means that we have pretty much nothing going on in this model t. It’s just like an empty model. We also have to specify this cutoff. This has to do with incremental cutoffs. We just have to specify any quality functions, so phys_equal is physical equality. You could also define your own equality function, and this is for comparing models to be clear. Next we have our type Action.t, and again, we’ve set this to unit. Yep.

I’m sorry, what’s the type of cutoff.

Oh, sorry. Cut off is… it takes two models and returns a Boolean.

Okay.

We’ve set Action to unit two, so basically we can’t really do anything in our app right now and we have this slightly confusing syntax. Basically this sexp_of generates a way to serialize our actions so that we can log them in our browser console. The function should_log allows us to specify which actions we want to log, and which ones we don’t. Here we’re logging everything. Next, we have the State module. This might be a little bit confusing because I had previously claimed that the model already represented the state of our app.

The idea here is that we want our model to be immutable, and we need that to be true mostly in order for everything to work as expected in Incr_dom. If we have any imperative states such as connections to servers, things like that, they should go in the state instead of the model. Next we provide an initial model. The type of our model is just unit so we provide a unit. We have our apply action in view functions. Apply action takes in a very boring action that can’t do anything and a model so we just return our model unchanged for now because the action wouldn’t be changing it anyways, and we have this view function that takes an incremental model.

Now, I’ve chosen to map on my model at the very top of this function. You’ll note that the model variable on the left of my equal sign is actually a constant value whereas on the right it’s an incremental value. Sorry, you can look at the bottom to see that. What this means is that below line 38, we can treat our model as a constant, we don’t have to worry about any incremental stuff but it also means that all the code below that line gets recomputed every single time our model changes. This is a simple way to not have to worry about incremental if you’re writing a very simple app, but if you’re worried about performance, this is just an absolutely terrible thing to do.

We’re going to leave it for now, and we’re going to come back to it later. I create a very simple virtual DOM node, a body with some text in it. Next, we have these three functions that I’m going to skip over. Note that you have to implement these, and we’ve included the trivial implementations here, but you have to include them in order to satisfy the Incr_dom app interface, and they are useful when you start writing more complicated apps, but for our case, we just don’t need them.

Now that we have our app module defined, all we have to do is pass it to Incr_dom along with the initial model we want to use, and we do this by passing it to this function Start_app.simple, and Incr_dom takes care of everything else for us. Just to make sure that we’re actually running the right code, I’m going to make a small change to this and see if that gets picked up in the browser. To compile this, we can use dune, which is a build system for OCaml and we just use this build command and specify what we want to be built. Of course we want our JavaScript file. This is very important. I’ve also included the HTML and the CSS so that they’ll all end up in the same build directory so that we can access them all in the same place.

This has successfully compiled. Now let’s look at our browser. If I refresh this, we now see My First Incr_dom App. Next, let’s make our app slightly more interesting. Let’s add an actual model to it. Let’s say we want to keep track of a list of counters. I’ll represent this using an int list. Next, I should actually provide an initial model that’s a list and not a unit and I’ll randomly generate this with length 100. All of the elements of this list will randomly have values between zero and 99.

Lastly, I want to actually display my model instead of just ignoring it. Just to note, the underscore before model tells the compiler that we are intentionally not using this variable, otherwise the compiler would complain to us. Now I do want to use it, and I’m going to create some row nodes. This List.mapi function basically goes over all of the elements in our list along with their indices. Here we have our index and our actual counter value. I’m going to create a row node with no attributes and with two children that are both cells. The first one will also have no attributes and will just display the index.

These correspond perfectly to HTML, so if you’re familiar with the tr tag, that’s what’s used for rows and the td tag is used for cells. I’m going to add one to the index, just so that it starts at one instead of zero and the second cell will also have no attributes and will display the counter value instead of the index. Lastly, we actually need to add this to our main virtual DOM node that we’re returning and I’m going to put it in a table. If I recompile this and refresh the page, now we see a table with two columns where the first column is just the indices and the second column is the randomly generated counter values. We have some kind of nice styling here. We have a border and alternating row colors. This was all done in the CSS file I showed you before.

Next, let’s add a bit of color to this. I’m going to color each of the counter values based on the actual value. Let’s say if our count is smaller than 10, we’ll make it green, otherwise, if it’s smaller than a hundred, we’ll make it blue, and otherwise we’ll make it red. It’ll start looking a bit scarier as the numbers get bigger. We’ll add this color to our second cell in the row by adding an attribute, and there’s this nice stuff function that takes a list of string pairs and my list is only going to contain one element where the property name is color and the value is the color that I computed above.

Now that we can compute styles and things like that using ordinary OCaml, and they can also be dynamic, they can change as a function of various other values in our virtual DOM nodes. If I compile this and refresh, now we see the colors that we expect. Smaller numbers are green, larger numbers are blue. We have no red because everything’s under a hundred, but that’s fine. Now let’s add some actions. I’m going to use a variant here. If you’re not familiar with variance, they basically express the idea that our actions will take one of several forms.

Let’s say we can have an increment action where we specify which index you want to increment and also a reset action where we also have to specify the index. Now that we have more interesting actions, let’s not ignore them, let’s apply them properly to our model. I’m going to introduce a little helper function here that will update a single index in our list. We pass it the index and the function to use to update it. We’ll map over our list and if we are at the target index then we’ll apply the function F to the counter value. If idx equals i then we apply the function and otherwise we just leave the value unchanged.

Now to actually handle our action, the way you need to usually handle variance is through pattern matching. A match statement looks something like this. If our action is an increment action, then we update our index by adding one to the current value. Otherwise, if it’s a reset action, then we update our index by ignoring the current value and just setting it to zero. Now, we have these actions. We are properly handling them if they’re ever scheduled, but we’re not actually scheduling anything. Let’s add some scheduling to our event handlers.

Now, if you’ll recall, I claimed that we’d have an argument called schedule here and instead we have this inject argument. Inject is pretty much the same as schedule, except it’s type has been slightly modified to force you to necessarily use it within the event handlers of your virtual DOM nodes, but otherwise it’s identical to schedule. Let’s add some event handlers to our rows. Let’s say if a user clicks once on a row that increments the counter. We inject an Action.Increment and we specify the row based on which row was clicked on.

Now, let’s do something similar with double clicking and resetting. If these are double clicks, then we inject a reset function instead. And just to make things a little bit more interesting, let’s also regularly schedule some randomly generated increment actions just to keep things ticking. We can do this in the on_start function, which I previously skipped over. This is a function that’s called once at the very beginning of initially running your app. First, let’s get our model life and let’s use that to randomly generate increment actions.

Here it will schedule something about 20 times a second. This should actually be every. We’ll call the schedule function and pass it around in the generated action. Action.Increment, the index will be a random number from zero to the length of the list minus one. Hopefully this compiles. Let’s see what that looks like in our browser. Now you see that some of the counters are incrementing on their own, and also if I click on a counter it’ll increment by one. If I double click, it goes to zero. Now we’ll see something turning red finally. This works pretty well. It’s pretty smooth. If I click or double click on a counter, it updates pretty quickly. Let’s see what happens if we make our list of counters much longer.

The first thing that I’m going to do is I’m going to add a length argument to our initial model. Now I’m going to change the implementation of that accordingly if I can find it. Now we actually pass in the length that we want. I’m going to copy this chunk of code that I don’t want to type out from scratch. Basically this looks at the query string of your URL and tries to parse the number of rows that you want out of it, and it looks for the parameter rows. If it doesn’t find this, it just defaults to 100, which we have down here.

Once we have this length, we actually need to pass it to our initial model. Let’s try this. If I set rows to 10, this indeed works. Now let’s just set this to a really, really big number and I happen to know that 30,000 is the number we want so that we start seeing everything break down, basically. You saw that took a few seconds to load and now that it’s loaded, if I try to scroll through, it’s extremely choppy. It’s scrolling up for some reason, and also if I try to reset a counter, you’ll notice quite a big lag before it actually updates. Let’s go back and fix this. Now as-

Is some of this because of the constant recalling of this data structure, or is it all just because of the… or it’s all just because of inherently [inaudible 00:47:56].

Sorry, can you repeat-

Well, some of what you’re doing there is using this data structure that you’re going through repeatedly. One could use [inaudible 00:48:06]-

Oh, interesting.

… there or-

That may make a difference but another thing that will make a difference is using incremental more effectively. In fact, I am going to change the list but not to an array. I’m going to change it to a map, and the reason for this is that once we are dealing with maps instead of lists, we can use that nice Incr_map library that I mentioned before. Let’s make this a map and also we have to specify the type of our keys. Let’s make our keys integers that are just the indices of the elements. Now everything breaks. As a preliminary attempt to fix this, I’m just going to change all calls from the list module to corresponding calls from the map module. That turns out to work quite well so I’m going to replace list with map.

That fixed a lot of things, clearly it didn’t fix everything. For one thing to initialize my map, I actually want you to create an association list first. I still want to use list, and I also want to now pass the index and use the index in the list I create and once I have this, I can pass it to a function in map that creates a map out of an association list, of_alist_ exn. Now we create our map. Some other issues are, even though map functions very closely correspond to list functions, some of them have some additional labels. I’m going to add those labels. This happens in two places and we have one last problem, which is that this row’s variable is now a map, but we actually need a list to pass into our Node.table function.

I just want the data in our map and I can get that using this Map.data function. Let’s test that this compiles. Looks like it does. However, note that we haven’t actually improved the efficiency of anything because we’re not actually using our Incr_map library yet. Let’s go in and do that. If you’ll recall, what I did initially here at line 46 is I mapped on my model at the very top of my function and this was nice because now I had this constant model value I could use. If we look at the bottom here, we can see it’s just a Model.t, and so that was easy to deal with, but it’s extremely inefficient because we are recomputing this whole chunk of code every single time even like the smallest thing changes in our model, which we’ve set to be at least 20 times second.

Let’s just remove this line. It’s a terrible line to have in there and now we have an incremental model. Oh, that was not what I hoped for. Yeah. We have a Model.t, Incr.t. If you’ll recall our model is a map, so that means we have a Map.t, Incr.t and that’s great because it means we can now use our Incr_map library. All I need to do is use the mapi function from our Incr_map library instead of the plain map function and everything almost works. There’s still an issue, which is now that our rows is an incremental value and we need a constant value to actually build our virtual DOM so I’m just going to map on our rows instead.

If I recompile it and refresh, notice that the page is still very slow to load. Recall that incremental helps with handling incremental changes. It doesn’t help with the initial work, but now that it’s loaded I can… well, hopefully I can scroll… sorry, one second. I can scroll a lot more smoothly and also if I click or double click on a counter, it updates much more quickly. Just one last thing that I want to add to this demo to show you the power of the Incr_map library is I want to keep track of the total over all of my counters.

I’m going to add this total variable and I’m going to compute it using another function in Incr_map called unordered_fold. What this does is you give it a starting value and a way to update that value for every element in your map and it gives you the total accumulated value after going through all of your elements. I want my starting value to be zero because this is a sum and I need to specify two functions. One of them is what to do if we have a new entry in our map and the other one is what to do if we remove an entry from our map.

I don’t care about the key, so I’ll just ignore it but every time we have a new entry, I want to add its data to our sum, and similarly, every time we remove an entry, I want to remove its data from our sum by subtracting it. Sum minus data. And now let’s actually use our total sum in our virtual DOM. If we refresh we hopefully now have…. Now we see this total at the top of our page. I guess it’s hard to tell if it’s actually computing the right value just from this example but one thing to note is it doesn’t make our program any more laggy. If I click or double click on something, it updates very efficiently still, and I can still scroll… well, more smoothly. I can still scroll fairly smoothly.

Actually, if we want to convince ourselves that this is doing reasonable things, we can just take a look at a much smaller number of rows. Let’s say 10 and here, for instance, if I reset everything to zero, we can see the total going down in a reasonable way. That was the demo. Let me close that. Basically we saw how to write this very simple Incr_dom app. Now, if we wanted to make this into a more realistic app, here’s some things we might need to consider or to add to it. First of all, you saw that the initial loading of our app was pretty slow. It took several seconds. It wasn’t great.

In practice one thing we would do is partial rendering, which I showed you in the trading systems GUI example, which is just the idea that you only actually really need to render the elements that are currently within your viewport and for elements that are outside your viewport, you can kind of just cheat and put in just a single empty element of the right size, and as long as you actually make sure it’s the right size, otherwise things jump around unexpectedly. This ends up working very nicely and is much more efficient. Another thing you might want to do is to keep certain elements scrolled into your viewport under certain circumstances.

This seems like a very specific thing to want, but I wanted to mention it because it ends up requiring the two functions that we’ve skipped over. I wanted to give you a sense of when you might need them. If we think back to the TS GUI example, we had this focus that even if we moved it to a row that was currently outside of the viewport, it would just be scrolled back into view. In order to do this, you need the two functions on display and update visibility.

Update visibility is called every time your DOM changes, or your windows resized or the user scrolls and this is your chance to make any measurements that you want, for instance, measuring where your focus is relative to the viewport. You need this in order to know by how much to scroll your page to get it back into view. Now on display is a function that’s called every time your DOM changes and what’s special about it is that it allows you to make decisions by comparing your current model to your previous model.

The way this would come in handy is let’s say we only want to scroll our focus back into view. If the current cell that the focus is on is different from the cell it was on in the previous model. If the user has just moved the cell we want to keep it scrolled into view. If they’re off doing something else completely unrelated to the focus, we want to leave it where it is because let’s say this scrolls to the top of the page to write some text in the search box, it would be pretty annoying for them to just forcibly be scrolled back to wherever the focus was.

Another thing that we often find ourselves wanting to do in more complex Incr_dom apps is to use incremental computations outside of our view function. A good example of this is again in the TS GUI where we have this set of rows and we sort them, and we sort them within an incremental computation because sorting them every time from scratch would be terribly inefficient. Now, once we throw in a focus, we also want to be able to figure out how to move the focus up by one row or down by one row. In our apply action function, we actually want to be able to access the list of sorted rows.

However, with our current interface if that happens in an incremental computation, it would have to happen within our view function and our apply action function doesn’t have access to anything that goes on in there so we have a problem. In fact, the API that I’ve been showing you so far is one of two available interfaces. It’s the simple interface. We also have a derived one available that achieves exactly this. It allows you to share incremental computations between various functions including your apply action and view function.

Hopefully now you have an idea of what it looks like to write a simple Incr_dom app and also some considerations when you go from a simple app to a more realistic one. If you circle back to the initial goals that I mentioned, we wanted to be able to easily write web UIs, we wanted for them to be easily maintainable, and we wanted to be able to optimize them. Thanks to js_of_ocaml, we can achieve the first two just from the fact that we can now write our apps in OCaml. Also thanks to the Elm architecture, it provides a very nice pattern that we can follow a very simple API and just like a very nice way of thinking about our UIs in general, so that really does make Incr_dom quite easy to use.

Thanks to some combination of virtual DOM and incremental and some other techniques that aren’t necessarily a part of the actual Incr_dom library like partial rendering, we’re able to optimize our apps so that even if they use really large amounts of data, they still perform very well. Ever since Incr_dom came into existence, we have been writing a lot more web UIs and it’s really made our lives a lot easier. Hopefully this talk has given you some insight into how Jane Street approaches writing web UIs.

One of the real benefits of React beyond just [inaudible 01:00:48] that API you guys have for [inaudible 01:00:55] model view was essentially the render function in React, was in React you can create high level [inaudible 01:01:02]. I was wondering if [inaudible 01:01:06] you can create higher level or higher level [inaudible 01:01:10] Incr_dom. My second question was sort of related. You compared this approach to for instance using React [inaudible 01:01:23] script [inaudible 01:01:25] that approach.

The first question about composability. There isn’t really an explicit concept of components. There will be soon. Jane is actually working on that, but just based on how the pattern is structured, it’s really easy to compose basically… to have components that follow that structure and to combine them into larger apps that also follow that structure. It’s not as explicit as in React, but it’s definitely something that’s easy and possible to do. You also asked me about comparing this approach to React with Reason and Buckles-

[crosstalk 01:02:02] using just React, which is like running things from OCaml to React. [inaudible 01:02:08] I’m just wondering if you have [inaudible 01:02:08] any comparisons between the two or is there any thoughts there?

I haven’t used any of these languages or frameworks, but I guess one nice thing about Incr_dom, is that because you get to build your own incremental app you have a lot of control over the performance of your app, you can really fine-tune it, and things like computing the total over all of your elements for instance, you can still do efficiently. I’m not familiar enough with React to know whether that’s possible, but I think Incr_dom adds some flexibility for certain types of things that you might want to do, and then I don’t know if you want me to go into BuckleScript but…. I mean, if you’re curious, why we use js_of_ocaml instead of BuckleScript, this is what I’m told. I don’t know firsthand, but I’ve heard that BuckleScript is harder to maintain so it ends up being a few versions behind the latest OCaml version and that’s the main reason we aren’t able to use it within Jane Street.

One really big benefit of it is that it produces actually readable JavaScript, which js_of_ocaml does not do, but it turns out we just don’t really care about that. We’re fine with giving that up.

First of all, that’s [inaudible 01:03:56] about web development on OCaml [inaudible 01:04:01] something. Can you say something about how Incr_dom compares to Elm… sorry, if I were an Elm developer, would I feel comfortable [inaudible 01:04:17] can you say something about the differences?

That’s a good question. I mean, a lot of the ideas are similar. I think in Elm, you’re not forced to use this Elm architecture. It just ends up kind of happening naturally, but it’s not enforced upon you. I don’t think that’s necessarily a disadvantage because it’s nice to just follow this pattern, not have to worry about how to structure your app. As far as the language I’ve heard that the Elm language is quite similar to OCaml in many ways. Beyond that, I guess I don’t know enough about Elm to get a sense of how easy it is to switch between the two but there’s certainly a lot of shared ideas between them.

In Elm they also have a concept of command or some exclusion for model [inaudible 01:05:09] except model in any way in the Incr_dom platform.

Sorry, could you repeat that?

So in Elm they also have a concept for command and some [inaudible 01:05:20] which are used to model the [inaudible 01:05:22] for getting subscription to somewhere else, just wondering is that also how [inaudible 01:05:32].

And this happens separately from updates, is that right?

Yes.

Okay.

[inaudible 01:05:38] use a model, and the other one is [inaudible 01:05:44] model and also a command which is [inaudible 01:05:48].

Interesting. I’m not really familiar with like an equivalent system within Incr_dom.

So [VDOM 01:06:02] doesn’t really have this analog of like Incr_map, right? Like whenever you go to the VDOM you’re like maybe optimizing instructions to sub trees, but then they always have a certain [inaudible 01:06:23] like Map.data, you can always subtract that whole list every time. So is it basically just the case that like at some point VDOM gets too big and then you’re like [inaudible 01:06:36] that tree and that like if it gets too big you need to do this… what do we call it, [crosstalk 01:06:37].

The partial library.

Is that the rule or are there other tricks that you either have or thought about or [inaudible 01:06:49]?

The thing you mentioned that every time you have to… if you have a map, you have to call Map.data on it and just completely reconstruct that. That’s something we have noticed and have been sad about. We haven’t really figured out a solution. I mean, you do save a lot of work by not actually having to reconstruct the individual nodes in the map every time, but yeah, that’s definitely a downside, and yeah, ultimately you do need to use other fancy techniques once your app becomes really, really big, like partial rendering but yeah. We don’t have a solution for that specific problem.

I’m not quite sure anymore but I slept on one of the slides, were there a modern state?

Yeah.

… [crosstalk 01:07:39]. How do you decide what goes into a state or what goes into your [inaudible 01:07:45]. I’m not sure, can you state at the end or…

Yes. I didn’t have state in the slides, I kind of ignored it. We had it in the coding demo and we did leave it empty. Generally, what’s important is that first of all, only our model can be used to generate our view. Any information we need in order to actually create our view has to be in the model. Second of all it’s important that the model’s immutable due to various things like how incremental works and things like that. Generally, if you have any state that is mutable such as connections to servers and stuff, you should put those in the state.

To show that [inaudible 01:08:32] people can code, did have other components like graphs [inaudible 01:08:37] something else which should also allow us to do, partial [inaudible 01:08:43].

That’s a great question. In fact, there’s been several UIs that have wanted to use graphs and it’s revealed that we definitely don’t have enough tooling around graphs specifically, unfortunately, but that is an area where we probably need to build up some components and I think it would be very useful.

Even if there is some overhead to using incremental computations, when do you find yourself reaching for an incremental model or when do you think it’s fine just to use a play model or just recompute the whole time?

That’s a great question. I don’t have a definite answer. I think there’s some experimentation involved until you become more familiar with the overhead cost. If you have a small model, even if you could improve it using incremental, sometimes that hurts more than it helps. Generally when your model is small and isn’t really laggy or performing worse than you’d hope without incremental, I guess there is no need to add an incremental. That’s a very good point. Yeah, there’s a lot of fiddling that you need to do sometimes to figure out if certain optimizations actually make things worse because of the overhead.

First of all, it’s amazing why you called it [inaudible 01:10:12] error, but in this regard how do you debug the results [inaudible 01:10:20]?

It’s a [inaudible 01:10:23] model.

JavaScript [crosstable 01:10:25].

JavaScript.

Sometimes [inaudible 01:10:28].

What kind of bugs do you mean exactly? I’ve never seen these bugs you speak of. No, I’m just joking. I mean, we’ve been talking a lot about-

[inaudible 01:10:43].

Yeah, so you can definitely log a lot of information to the console. We’ve been talking about putting together some testing frameworks. That’s definitely an area we need to work on. We’ve considered even just looking at the virtual DOM turning it into HTML and inspecting that. You can also just test it by running it and seeing if it does what you want, but yeah, I guess we could have better tools in place that we have been thinking about, but haven’t really put together.

You were involved, and it’s hardly talking about the client server end.

Yeah. That’s right.

And not at all about the server side or how to [inaudible 01:11:28] application tool. I presume that you have tools for that as well, I mean given that you’re writing in the same language on both sides if not on the same line, and in fact on the same object [inaudible 01:11:41] it would compile.

Yeah. I guess one thing that we have that makes it really easy to talk to the server is we have RPCs over web sockets. As far as fancier tools than that we have some ideas of best practices. For instance, we’ve noticed that if you use pipe RPCs that just keep giving you updates regardless of whether the browser is able to process them that causes the browser to crash, so sometimes polling is a better alternative, things like that. I guess I’m not too familiar with other tools that we have. For my purpose that has just been sufficient.

[inaudible 01:12:33] generate some requests to the server. It’s kind of a similar question about side effects and it’s going to take a while, right? Have some [inaudible 01:12:43] JavaScript callback, right? Can you pull that inject function from callback or how do you handle that situation?

What you can do is you can schedule an action immediately and when you… Sorry, let me think. In your apply action function, you were given this schedule action function that you can then use in asynchronous computations to schedule actions later on. That’s fairly easy to handle, and it is something we do a lot.

Whenever you schedule an action, if that process takes a while, the page is told they [inaudible 01:13:33] schedule all their actions, right?

Yeah.

Are there any guarantees about how the actions that get applied to the model actually are the actions that got generated by the same model, I suppose, to actions generated by an older version?

Yeah, that’s a great question. You always have this problem if you schedule actions asynchronously that by the time you actually get to your model to apply the action, it might have completely changed from the time you tried to schedule the action. Yeah, that’s a problem that you have to deal with. Often you should do checks on your model before applying those actions to make sure the action still makes sense to apply, things like that. Yeah, that’s a good point.

Thank you.