Skip to main content

Future of Coding Weekly 2020/08 Week 5

For some reason tinyletter decided to not publish the newsletter in the archive so I'm posting it here.

If you want to subscribe to the newsletter, it's here: https://tinyletter.com/marianoguerra/

Subtext 1 Demo, Layered Text, VR Visual Scripting, Automated Game Design, Dynamic Sketching in AR, Tiny Structure Editors for Low, Low Prices & more

Two Minute Week

🎥 This Week in Instadeq: Event Triggers via Mariano Guerra

🧵 conversation

This week I added support for Event Triggers, a way to react to changes and do things on other entities

Share Our Work

💬 Chris Rabl

🧵 conversation

I've been doing more and more writing lately, and have been wishing for a tool that allows me to write my outlines, drafts, and final compositions in the same editor window with the ability to toggle any of those "layers" on and off at will, merge them, copy selections to new layers, etc. It would work sort of like Photoshop but for writing... I have a feeling these principles could also work as an IDE extension (imagine being able to hide the "code" layer and show only the "comments" layer, or the "documentation" layer). Curious to hear your thoughts, or whether anyone else is working on something similar?

🎥 layered text

📝 Using Gizmos via Scott Anderson

🧵 conversation

A year ago I was working on VR Visual Scripting in Facebook Horizon. They've recently started to share some more information leading up to Facebook Connect. I figured the scripting system would either be largely the same, or entirely rewritten since I left. It seems like it's mostly in tact based on documentation shared

🎥 Create and Share Interactive Visualizations from SpaceX's JSON API and 🎥 Create and Share Visualizations of Premier League Matches from a CSV via Mariano Guerra

🧵 conversation

📝 root via Dennis Heihoff

🧵 conversation

What started with me reverse engineering notion became a data-first recursive UI resolver I called root.

Here's how it differs from most common technologies today:

  • Approaches to UI development like react.js + graphQL require UI components to request data in a shape that satisfies the UI tree. This means the shape of the data is determined by the UI tree. Root takes an inverse approach where the UI tree is determined by the shape of the data.
  • A major benefit of this approach is that the UI layout is thus entirely determined by data, data that can be traversed, transformed and stored in arbitrary ways and by arbitrary means.
  • This is powerful for unstructured, user-determined, block-based UI's like rich documents (think Roam Research, Notion etc.) enabling queries and functions that, based on users' demands, derive the optimal presentation of a document.

It packs a few more punches. The best example is probably this (in about 200 LoC).

Thinking Together

📝 model of computation via Nick Smith

🧵 conversation

Why isn't any kind of logic programming considered a model of computation? Why do we talk about Turing Machines and recursive functions as fundamental, but not inference? I can't find any resources discussing this disparity. It's like there are two classes of academics that don't talk to each other. Am I missing something?

📝 Motoko, a programming language for building directly on the internet - Stack Overflow Blog via Mike Cann

🧵 conversation also discussed here 🧵 conversation

Anyone played with Motoko yet? looks really interesting, kind of reminds me of Unison in some ways

📝 https://twitter.com/cmastication/status/1299366037402587137?s=21 via Cameron Yick

🧵 conversation

Pondering: how important is it for a making environment to be made from the same medium you’re making with if your main goal isn’t making interfaces? The Jupyter ecosystem has come quite far despite relatively few people using it to write JS: https://twitter.com/cmastication/status/1299366037402587137?s=21

🐦 JD Long: Observation from Jupyter Land: The Jupyter ecosystem has a big headwind because the initial target audience for the tool (Julia, Python, R) has a small overlap with the tool/skills needed to expand the ecosystem, namely Javascript.

That's not a criticism, just an observation.

💬 Hamish Todd

🧵 conversation

In the thing I am making, you can't have a variable without choosing a specific example value for that variable. This is surely something that's been discussed here before since Bret does it in Inventing On Principle. What do folks think of it?

Content

📝 Tiny Structure Editors for Low, Low Prices! via Jack Rusher

🧵 conversation

Fun paper from 2020 IEEE Symposium on Visual Languages and Human-Centric Computing

🎥 Subtext 1 demo (from 2005) via Shalabh Chaturvedi

🧵 conversation

Jonathan Edwards recently uploaded the Subtext 1 demo (from 2005).

It has a lot of interesting takes - and most (all?) that I agree with. E.g. edit time name resolution, debugging by inspection, a concrete model of time, inline expansion of function calls, and more.

📝 It's the programming environment, not the programming language via Ope

🧵 conversation

“But while programming languages are academically interesting, I think we more desperately need innovation in programming environments.

The programming environment isn’t a single component of our workflow, but the total sum enabled by the tools working together harmoniously. The environment contains the programming language, but also includes the debugging experience, dependency management, how we communicate with other developers (both within source code and without), how we trace and observe code in production, and everything else in the process of designing APIs to recovering from failure.

The story of programming language evolution is also a story of rising ideas in what capabilities good programming environments should grant developers. Many languages came to popularity not necessarily based on their merits as great languages, but because they were paired with some new and powerful capability to understand software and write better implementations of it.”

🎥 Getting Started in Automated Game Design via Scott Anderson

🧵 conversation

Mike Cook has done a lot of research into automated game generation. He recently released this video which is both a tutorial and an overview of the field.

📝 Gatemaker: Christopher Alexander's dialogue with the computer industry via Stefan Lesser

🧵 conversation

Don’t read this for the application “Gatemaker”. Read this for a fascinating outsider’s view on the software industry, systems design, and end-user programming.

🎥 RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching via Jack Rusher

🧵 conversation

I really like this new AR work from Ryo Suzuki, et al.

A tour through the beam ADT representation zoo

The Languages

Dynamically Typed

Statically Typed

Column meaning

Record Type

named set of fields (key, value pairs), also refered as types, structs, records etc

Union Type

like a record type but with more than one "shape", also refered as discriminated unions, variants etc

Type Dispatch

a function that will have different implementations according to the type of one (or more) of its arguments, also refered as protocols or multi methods

TL;DR Table

Language

Inspiration

Record Type

Union Type

Type Dispatch

Dynamic

Clojerl

Clojure

Yes

No

Yes

Efene

Python/JS

Yes*

No*

No

Elixir

Ruby

Yes

No*

Yes

Erlang

Prolog

Yes*

No*

No

LFE

Common Lisp

Yes*

No*

No

Static

Alpaca

ML

Yes

Yes

No*

Fez

F#

Yes

Yes

Yes

Gleam

ML/Rust

Yes

Yes

Not Yet?

Hamler

Purescript

Yes

Yes

Yes

PureErl

Purescript

Yes

Yes

Yes

Groups

Languages that compile record types to erlang records

Languages that compile record types to erlang maps

  • Alpaca: __struct__ field for records, no extra field for anonymous records

  • Clojerl: __type__ field

  • Elixir: __struct__ field for structs

  • Purerl: no extra field for anonymous records

  • Hamler : no extra field for anonymous records

Languages with union types

  • Alpaca: tagged tuple if it has values, atom if not

  • Gleam: tagged tuple if it has values, atom if not

  • Fez: tagged tuple if it has values, atom if not

  • Purerl: tagged tuple (not sure about variants with no values, should be like Hamler)

  • Hanler: tagged tuple even with variant with no values

Languages that do type dispatch

Notes

Alpaca

Records: "anonymous records".

Records are compiled as maps with the KV '__struct__' => 'record'. Because Alpaca doesn't provide any reflection facilities, more type information isn't propagated to the generated Core Erlang.

In the tag (in the case of variants/discriminated unions), it's just the atom representation of the tag name itself.

E.g. Some_tag 1 gets compiled to {'Some_tag', 1}

Alpaca's records get an extra key-value pair of '__struct__' to a rough description of its type/structure if the "structure" is flagged as a record.

Tagged Unions: (variants in OCaml) get compiled as an atom if there's no associated value, and as a tuple if there is.

Type Dispatch: Author says: "Not currently. This sort of thing might get handled by type classes but I haven't gone too far down that line of thinking yet"

Clojerl

Records: (deftype) compiled to Erlang maps with a special __type__ field.

Tagged Unions: No

Type Dispatch: defprotocol and deftype, extend-type, extend-protocol work as in Clojure.

Protocols are not reified as in Clojure, there are no runtime protocol objects.

Elixir

Records: structs are compiled to Erlang maps with a special __struct__ field.

Tagged Unions: No (usually ad-hoc tagged tuples are used for this)

Type Dispatch: Protocols are collected and consolidated at compile time

Gleam

Records: Compiled to Erlang records (hrl files are generated)

Tagged Unions: Compiled to tagged tuples, they are just gleam custom types with multiple "constructors", if a variant has no values it gets compiled to an atom

Type Dispatch: Not Yet? Will Gleam have type classes?

Fez

Records: Compiled to Erlang records

Tagged Unions: Compiled to tagged tuples

Type Dispatch: Class method calls

LFE, Efene and Erlang

LFE and Efene are just "dialects" of Erlang, that's why they are covered together here.

Records: Erlang records, which are compiled to a tuple where the first value is an atom with the name of the record: LFE Records, Efene Records, Erlang Records

Tagged Unions: Since they are dynamically typed they can use tagged tuples for this, there's no need to declare them, examples are functions that return {ok, V} or {error, Reason}.

Type Dispatch: No

PureErl

Records: Compiled to Erlang maps (without an extra field), really similar to alpaca "anonymous records"

Tagged Unions: Compiled to tagged tuples

Type Dispatch: Type Classes

There are also newtype

Hamler

Records: Compiled to Erlang maps (without an extra field)

Tagged Unions: Compiled to tagged tuples

Type Dispatch: Type Classes

Why I don't like concept cars addendum

I don't like to glorify Steve Jobs, but sometimes he expresses ideas the right way and the fact that he, unlike me, showed clearly that those ideas can deliver good results may show that those ideas are valid.

Here's a part of an interview from him (emphasis mine):

... they have no conception of the craftsmanship that’s required to take a good idea and turn it into a good product and they really have no feeling in their hearts usually about wanting to really help the customers there’s a just a tremendous amount of crafts ship in in-between a great idea and a great product and as you evolved that great idea it changes and grows it never comes out like it starts because you learn a lot more as you get into the subtleties of it and you also find there’s tremendous trade-offs that you have to make I mean you know there are there are just certain things you you can’t make electrons do there are certain things you can’t make plastic do or glass do or factories do or robots do and as you get into all these things designing a product is keeping 5,000 things in your brain these concepts and fitting them all together in in kind of continuing to push to fit them together in new and different ways to get what you want and every day you discover something new that is a new problem or a new opportunity to fit these things together a little differently and it’s that process that is the magic.

From here:

Two types of software prototypes and why I don't like concept cars

When I was a child I used to love concept cars, I loved how they were much better than regular cars.

I would listen carefully to when they would be available in the market, most of the time that information wasn't mentioned, sometimes the date was so far in the future that it wasn't interesting, by that time we would already have flying cars!

With time I started noticing that none of the concept cars from my childhood were on the streets, not even close. Worst was when a really bad version of them was available for sale, the worst deception ever.

Now when I see a company showing off with a concept car I think the opposite, that company is running out of real ideas or has lost the ability to execute novel designs, and tries to justify it by showing shiny things that it knows will never see the light of day.

Why I'm talking about concept cars? well, because there are different types of concept cars and different types of software prototypes, but they are almost the same.

The prototypes that require you to accept things that are never going to be feasible but are sold as if they are possible are the worst. Either because they violate basic laws of physics, materials, safety, regulations, performance, user experience or just because they focus on a single concept by disregarding others that are required when the thing migrates from prototype to production.

If you are upfront and tell that the prototype is an exploration of "what if we push this dimension to the extreme", I'm ok with it, it's a learning experience, you may learn a lot about that dimension and how it relate to others, what are its limitations and so on.

But the prototype should be clearly marked as such.

The other useful prototypes are as learning exercises, I like to build throw away prototypes as my first approach to something, as a way to learn more about something and have a better idea for next time. You should also mark them clearly as such.

The third useful prototype is the initial stage of something you want to grow into a product but want to show the potential along the way, you may throw some versions away in the early stages because you learned something that required a big reformulation and it's easier to start from scratch than to refactor it.

But from the beginning the prototype should be grounded in reality, what's possible and how the main concept relates to other features that may be a year or more in the future but they are going to be required for the prototype to turn into a product.

You can't have a slow/complex prototype in the early stages unless you have a clear idea on how it's going to get faster/simpler, small optimizations here and there aren't going to cut it once the performance/complexity penalty of all the extra features starts creeping up. You have to think up front how those other features will fit once you reach that point.

Of course, if you are building something new, at some point you will be in a new territory and some things may require some reformulation, that's good, but at that point you need to have buffers of performance and other metrics and a simple core architecture and code to be able to solve those problems, even if not in the optimal way. That's why you built this new thing, to push the boundary a little further.

If you get the chance to build a new prototype after the current one turned into a finished product, you can then incorporate those new insights at the beginning to be able to push a little further than before. Rinse and repeat.

What you should not do is to build a prototype that starts to fall apart even when only solving the single problem you care about and call the remaining just an implementation detail or "left as an exercise for the engineers".

It may sound like a contradiction, but to build good prototypes you have to be good at building complete products, otherwise you can't guide your initial design on constraints that you never experienced. A prototype should eventually be the foundation of something much bigger and complex, you can't build that on unstable foundations. If not you, somebody on the team must provide the experience that comes from completing, polishing and maintaining something that survives the contact with reality.

As Mike Tyson said: "Everybody has a plan until they get punched in the mouth."

Or put another way:

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system.

—John Gall

There's only one dimension you can ignore if you have the time/money to do so, and that's price: see the Experience curve effect. Just don't be too early ;)

inb4 appeal to visionaries, check Vannevar Bush's memex, it was a full design, it was easy to see how you could build it, even when some technologies were in the future, everything was feasible. Ivan Sutherland Sketchpad Demo was a complete runnable thing, same with Douglas Engelbart's Mother of all Demos, Xerox PARC's Alto and Bret Victor's Dynamicland.

Don't fool others, but most important, don't fool yourself.

On utopians and the fact that software must exist and solve real problems

Yet a man who uses an imaginary map, thinking that it is a true one, is likely to be worse off than someone with no map at all; for he will fail to inquire whenever he can, to observe every detail on his way, and to search continuously with all his senses and all his intelligence for indications of where he should go

—E. F. Schumacher, Small is Beautiful

There's an idea I first saw at Django Conf that I really liked, the opening keynote is called "Django Sucks" and it's about everything that is wrong with Django.

Since then I've been promoting the idea on every conference I've been and many times I fantasize about giving the "X Sucks" talk myself.

I'm part of more than one community in the utopian spectrum, 2 weeks ago I gave a talk at Bob Konf titled Programming by any other name where I showed how the future is already here, we just have to find it and help its creators gain adoption, here's the other side of that talk:

As a temporary member of the software utopians, an activity we share is to imagine alternative realities in which every wrong decision was instead replaced with the "right" one, with the benefit of hindsight, of course.

The word "right" in quotes because it may have yet to be tested in the real world and may require the suspension of disbelief on many aspects of reality. We may call this activity "counterfactual porn".

I say temporary member of the software utopians because I'm also a member of the software builders, people that build software that real users are willing to use (and pay for) to solve real problems.

One of the utopians' hobbies is to look down on systems that exist and solve real problems, pointing at its flaws.

These flaws are in contrast to systems that only exist as an idea, paper, or at most as a small prototype that proves a single point and solves a single problem carefully crafted to make it shine.

But there's a gap that utopians never seem to cross.

The gap from early prototype that at most proves a point or shows a new idea, to the point where that single thing is part of a whole that people can use to solve a large variety of complex problems.

Some utopians dared to cross the gap, General Magic, Pharo, Racket, Genera to name a few.

These Pioneers turned settlers faced the fact that "If you build it, they will come" is usually not true.

Those products were (and some are) things that you can get and use, but for some reason they failed to catch on.

Here utopians usually appeal to the "No true Scotsman fallacy", if not to the simpler "people are stupid/don't know what's good for them" and go back to the confortable possition of throwing stones at things others build and maintain.

When someone takes the failed idea (or prototype) and adapts it into something that people actually use, taking into account the limitations of reality, society, economics and people's behaviors, the utopians proceed to complain about how the new thing is a bastardized version of the original idea and how if they were to do it they would stay true to the original.

The exercise of actually doing either is almost never attempted, or is attempted and quickly abandoned.

In some cases it's completed and for "strange" reason it fails to gain adoption. GOTO No true Scotsman/People are stupid.

The implementation, its users or some aspect of reality are always blamed, the idea must be kept untouched.

I usually go around preaching ideas by thinkers I admire, I think it's really useful to consider them to improve the systems that builders create.

Pure Ideas are important, people stopping at prototypes too, but as much as I believe we need to learn more from history and study great thinkers, researchers and their ideas, I also think we should at least respect and listen to people building finished products that have to face the harsh reality of productive usage of working software.

While we are at it we could also listen to the users of such systems, which we love to talk about, but almost never talk to, let alone let them tell us something that may shape our ideas.

Ideas are only impactful when they get turned into things that real people can use to solve real problems.

During this process, pure ideas have to be adapted at each step, the end result is usually not as pure as one would wish, but that's the price to pay to get from idea to reality.

This involves at least 3 kinds of people, Utopians/Pioneers, Builders/Settlers and Users/Citizens, you need the 3 to collaborate, communicate and respect each other's roles and constraints.

If you are of one kind and think you could do a better job at being the other, then before telling them, try showing them, all the way. You may learn that it's not that easy.

If you don't want to show them, then collaborate and listen, you may learn something new that may improve the chances your idea gets adopted.

The process from idea to adoption also involves 3 different timescales, short, middle and long term (similar to operational, tactical and strategic levels of planning).

Execution has to work at each level individually but to achieve the long term vision, the short and middle term need to be aligned with the long term, even if in between it has to take some detours/shortcuts.

When the 3 roles collaborate and are willing to adapt the idea to consider each other's constraints and plan for the 3 timescales as a whole, they may have a better change to achieve the utopian objective, even when during the process it doesn't look like it.

The alternative is to stay forever at the idea level complaining at people trying to bring it to reality.

PS: I may not be only talking about software

RFC: Elixir Module and Struct Interoperability for Erlang

This is a RFC in the shape of a project that you can actually use, I'm interested in your feedback, find me as marianoguerra in the erlang and elixir slacks and as @warianoguerra on twitter.

The project on github: https://github.com/marianoguerra/exat and on hex.pm: https://hex.pm/packages/exat

See the exat_example project for a simple usage example.

Here's the description of the project:

Write erlang friendly module names and get them translated into the right Elixir module names automatically.

The project is a parse transform but also an escript to easily debug if the transformation is being done correctly.

Erlang Friendly Elixir Module Names

A call like:

ex@A_B_C:my_fun(1)

Will be translated automatically to:

'Elixir.A.B.C':my_fun(1)

At build time using a parse transform.

The trick is that the @ symbol is allowed in atoms if it's not the first character (thank node names for that).

We use the ex@ prefix to identify the modules that we must translate since no one[1] uses that prefix for modules in erlang.

Aliases for Long Elixir Module Names

Since Elixir module names tend to nest and be long, you can define aliases to use in your code and save some typing, for example the following alias declaration:

-ex@alias(#{ex@Baz => ex@Foo_Bar_Baz,
            bare => ex@Foo_Long}).

Will translate ex@Bar:foo() to ex@Foo_Bar_Baz:foo() which in turn will become 'Elixir.Foo.Bar.Baz:foo()

It will also translate the module name bare:foo() into ex@Foo_Long:foo() which in turn will become Elixir.Foo.Long:foo()

Creating Structs

The code:

ex:s@Learn_User(MapVar)

Becomes:

'Elixir.Learn.User':'__struct__'(MapVar)

The code:

ex:s@Learn_User(#{name => "bob", age => 42})

Becomes:

'Elixir.Learn.User':'__struct__'(#{name => "bob", age => 42})

Which in Elixir would be:

%Learn.User{name: 'bob', age: 42}

Aliases in Structs

The following alias declaration:

-ex@alias(#{ex@struct_alias => ex@Learn_User}).

Will expand this:

ex:s@struct_alias(#{name => "bob", age => 42})

Into this:

'Elixir.Learn.User':'__struct__'(#{name => "bob", age => 42})

Pattern Matching Structs

Function calls are not allowed in pattern match positions, for example on function/case/etc clauses or the left side of a =, for that there's a different syntax:

get_name({ex@struct_alias, #{name := Name}}) ->
    Name;
get_name({ex@struct_alias, #{}}) ->
    {error, no_name}.

Becomes:

get_name(#{'__struct__' := 'Elixir.Learn.User', name := Name}) ->
    Name;
get_name(#{'__struct__' := 'Elixir.Learn.User'}) ->
    {error, no_name}.

And:

{ex@struct_alias, #{name := _}} = ex:s@Learn_User(#{name => "bob", age => 42})

Becomes:

#{'__struct__' := 'Elixir.Learn.User', name := _} =
        'Elixir.Learn.User':'__struct__'(#{name => "bob", age => 42})

This is because that pattern will match maps that also have other keys.

Note on Static Compilation of Literal Structs

On Elixir if you pass the fields to the struct it will be compiled to a map in place since the compiler knows all the fields and their defaults at compile time, for now exat uses the slower version that merges the defaults against the provided fields using 'Elixir.Enum':reduce in the future it will try to get the defaults at compile time if the struct being compiled already has a beam file (that is, it was compiled before the current file).

Use

Add it to your rebar.config as a dep and as a parse transform:

{erl_opts, [..., {parse_transform, exat}, ...]}.
...
{deps, [exat, ...]}

Build

To build the escript:

$ rebar3 escriptize

Run

You can run it as an escript:

$ _build/default/bin/exat pp [erl|ast] path/to/module.erl

For example in the exat repo:

$ _build/default/bin/exat pp erl resources/example1.erl
$ _build/default/bin/exat pp ast resources/example1.erl

Syntax Bikesheding

The syntax I chose balances the need to not produce compiler/linter errors or warnings with the objective of avoiding accidentally translating something that shouldn't be translated.

Please let me know what you think!

[1] Famous last words

Riak Core on Partisan on Elixir Tutorial: Resources

A List of resources related to riak_core and partisan.

Riak Core

Project

  • riak_kv: Riak KV itself

  • riak_pg: Distributed process groups with riak_core

  • dalmatinerdb: A fast, distributed metric store

  • riak_test_core: riak_test fork which refactors riak_test to not bo targeted directly towards riak_kv and makes it more library like

  • nkdist: a library to manage Erlang processes evenly distributed in a riak_core cluster

  • riak_id: A clone of Twitter's Snowflake, built on riak_core

  • DottedDB: A prototype of a Dynamo-style distributed key-value database, implementing Server Wide Clocks as the main causality mechanism across the system

  • riak_pipe: riak_pipe allows you to pipe the output of a function on one vnode to the input of a function on another

Docs

Partisan

Partisan.cloud: Partisan Website

Projects

  • Lasp: Lasp is a suite of libraries aimed at providing a comprehensive programming system for planetary scale Elixir and Erlang applications

  • Vonnegut: an append-only log that follows the file format and API of Kafka 1.0

  • Erleans: Erlang Orleans

Riak Core on Partisan on Elixir Tutorial: Migrating Data with Handoff

Previous post: Riak Core on Partisan on Elixir Tutorial: We can make a Key Value Store out of that.

We are going to implement handoff by completing some of the remaining callbacks in our vnode.

Why? if a node dies and another takes his work or if we add a new node and the vnodes must be rebalanced we need to handle handoff.

The reasons to start a handoff are:

  • A ring update event for a ring that all other nodes have already seen.

  • A secondary vnode is idle for a period of time and the primary, original owner of the partition is up again.

When this happen riak_core will inform the vnode that handoff is starting, calling handoff_starting, if it returns false it’s cancelled, if it returns true it calls is_empty, that must return false to inform that the vnode has something to handoff (it’s not empty) or true to inform that the vnode is empty, in our case we ask for the first element of the ets table and if it’s the special value $end_of_table we know it’s empty, if it returns true the handoff is considered finished, if false then a call is done to handle_handoff_command.

The fold_req_v2 macro is defined in the riak_core_vnode.hrl header file as riak_core_fold_req_v2 which we include using the Record module.

This function must iterate through all the keys it stores and for each of them call foldfun with the key as first argument, the value as second argument and the latest acc_in value as third.

The result of the function call is the new acc_in you must pass to the next call to foldfun, the last accumulator value (acc_final) must be returned by handle_handoff_command.

For each call to foldfun(k, v, acc_in) riak_core will send it to the new vnode, to do that it must encode the data before sending, it does this by calling encode_handoff_item(k, v), where you must encode the data before sending it.

When the value is received by the new vnode it must decode it and do something with it, this is done by the function handle_handoff_data, where we decode the received data and do the appropriate thing with it.

When we sent all the key/values handoff_finished will be called and then delete so we cleanup the data on the old vnode.

You can decide to handle other commands sent to the vnode while the handoff is running, you can choose to do one of the following:

  • Handle it in the current vnode

  • Forward it to the vnode we are handing off

  • Drop it

What to do depends on the design of you app, all of them have tradeoffs.

A diagram of the flow:

+-----------+      +----------+        +----------+
|           | true |          | false  |          |
| Starting  +------> is_empty +--------> fold_req |
|           |      |          |        |          |
+-----+-----+      +----+-----+        +----+-----+
      |                 |                   |
      | false           | true              | ok
      |                 |                   |
+-----v-----+           |              +----v-----+     +--------+
|           |           |              |          |     |        |
| Cancelled |           +--------------> finished +-----> delete |
|           |                          |          |     |        |
+-----------+                          +----------+     +--------+

Replace the content of lib/civile_vnode.ex with:

defmodule Civile.VNode do
  require Logger
  @behaviour :riak_core_vnode

  require Record
  Record.defrecord :fold_req_v2, :riak_core_fold_req_v2, Record.extract(:riak_core_fold_req_v2, from_lib: "riak_core/include/riak_core_vnode.hrl")

  def start_vnode(partition) do
    :riak_core_vnode_master.get_vnode_pid(partition, __MODULE__)
  end

  def init([partition]) do
    table_name = :erlang.list_to_atom('civile_' ++ :erlang.integer_to_list(partition))

    table_id =
      :ets.new(table_name, [:set, {:write_concurrency, false}, {:read_concurrency, false}])

    state = %{
      partition: partition,
      table_name: table_name,
      table_id: table_id
    }

    {:ok, state}
  end

  def handle_command({:ping, v}, _sender, state = %{partition: partition}) do
    {:reply, {:pong, v + 1, node(), partition}, state}
  end

  def handle_command({:put, {k, v}}, _sender, state = %{table_id: table_id, partition: partition}) do
    :ets.insert(table_id, {k, v})
    res = {:ok, node(), partition, nil}
    {:reply, res, state}
  end

  def handle_command({:get, k}, _sender, state = %{table_id: table_id, partition: partition}) do
    res =
      case :ets.lookup(table_id, k) do
        [] ->
          {:ok, node(), partition, nil}

        [{_, value}] ->
          {:ok, node(), partition, value}
      end

    {:reply, res, state}
  end

  def handoff_starting(_dest, state = %{partition: partition}) do
    Logger.debug "handoff_starting #{partition}"
    {true, state}
  end

  def handoff_cancelled(state = %{partition: partition}) do
    Logger.debug "handoff_cancelled #{partition}"
    {:ok, state}
  end

  def handoff_finished(_dest, state = %{partition: partition}) do
    Logger.debug "handoff_finished #{partition}"
    {:ok, state}
  end

  def handle_handoff_command(fold_req_v2() = fold_req, _sender, state = %{table_id: table_id, partition: partition}) do
    Logger.debug "handoff #{partition}"
    foldfun = fold_req_v2(fold_req, :foldfun)
    acc0 = fold_req_v2(fold_req, :acc0)
    acc_final = :ets.foldl(fn {k, v}, acc_in ->
        Logger.debug "handoff #{partition}: #{k} #{v}"
        foldfun.(k, v, acc_in)
    end, acc0, table_id)
    {:reply, acc_final, state}
  end

  def handle_handoff_command(_request, _sender, state = %{partition: partition}) do
    Logger.debug "Handoff generic request, ignoring #{partition}"
    {:noreply, state}
  end

  def is_empty(state = %{table_id: table_id, partition: partition}) do
    is_empty = (:ets.first(table_id) == :"$end_of_table")
    Logger.debug "is_empty #{partition}: #{is_empty}"
    {is_empty, state}
  end

  def terminate(reason, %{partition: partition}) do
    Logger.debug "terminate #{partition}: #{reason}"
    :ok
  end

  def delete(state = %{table_id: table_id, partition: partition}) do
    Logger.debug "delete #{partition}"
    true = :ets.delete(table_id)
    {:ok, state}
  end

  def handle_handoff_data(bin_data, state = %{table_id: table_id, partition: partition}) do
    {k, v} = :erlang.binary_to_term(bin_data)
    :ets.insert(table_id, {k, v})
        Logger.debug "handle_handoff_data #{partition}: #{k} #{v}"
    {:reply, :ok, state}
  end

  def encode_handoff_item(k, v) do
     Logger.debug "encode_handoff_item #{k} #{v}"
     :erlang.term_to_binary({k, v})
  end

  def handle_coverage(_req, _key_spaces, _sender, state) do
    {:stop, :not_implemented, state}
  end

  def handle_exit(_pid, _reason, state) do
    {:noreply, state}
  end

  def handle_overload_command(_, _, _) do
    :ok
  end

  def handle_overload_info(_, _idx) do
    :ok
  end
end

Stop all nodes if you have them running.

Let's clean all the data on each node and the ring state so that we can join them again each time we want to try handoff, to make it simpler to run the commands and remember them, put this in a file called Makefile at the root of the project, make sure the indented lines are indented with tabs and not spaces:

start_single:
   iex --name dev@127.0.0.1 -S mix run

start_node1:
   MIX_ENV=dev1 iex --name dev1@127.0.0.1 -S mix run

start_node2:
   MIX_ENV=dev2 iex --name dev2@127.0.0.1 -S mix run

start_node3:
   MIX_ENV=dev3 iex --name dev3@127.0.0.1 -S mix run

clean:
   rm -rf data_1 data_2 data_3 data log ring_data_dir*

setup:
   mix deps.get

The project at this stage is available in the tag handoff-1

Now let's clean the date to start from scratch:

make clean

On a new terminal:

make start_node1

Inside iex run:

for i <- :lists.seq(1, 100) do
    Civile.Service.put("k#{i}", i)
end

This will insert k1: 1, k2: 2,..., k100: 100 , since we are running a single node they will all go to vnodes in node1.

On a new terminal:

make start_node2

Inside iex run:

for i <- :lists.seq(101, 200) do
    Civile.Service.put("k#{i}", i)
end

This will insert k101: 101, k102: 102,..., k200: 200 , since we haven't joined node2 to node1, they will all go to vnodes in node2.

On a new terminal:

make start_node3

Inside iex run:

for i <- :lists.seq(201, 300) do
    Civile.Service.put("k#{i}", i)
end

This will insert k201: 201, k202: 202,..., k300: 300 , since we haven't joined node3 to node1 or node2, they will all go to vnodes in node3.

Let's join node2 and node3 to node1 to form a cluster, on node2 and node3 run:

:riak_core.join('dev1@127.0.0.1')

You will see handoff transfering vnodes and their data between nodes and stopping the vnodes in the old nodes when it finishes transfering their data and removing it.

Periodically you can check the status with:

{:ok, ring} = :riak_core_ring_manager.get_my_ring
:riak_core_ring.pretty_print(ring, [:legend])

Riak Core on Partisan on Elixir Tutorial: We can make a Key Value Store out of that

Previous post: Riak Core on Partisan on Elixir Tutorial: Getting Started.

Now that we have the foundation of our service laid out, let's make it do something useful.

Given that riak_core is an implementation of the Dynamo Architecture which was created to build scalable key value stores, we should follow the same path and build one.

Let's start from the end, imagine our code is ready, what would be the interaction with it?

Let's imagine it's like this, first we ask the service to get the value for a key that doesn't exist yet:

iex(dev@127.0.0.1)1> Civile.Service.get(:k1)
{:ok, :"dev@127.0.0.1",
 913438523331814323877303020447676887284957839360, nil}

We get a tuple as response with 4 items:

  1. status = :ok

    • The request was handled correctly

  2. node = :"dev@127.0.0.1"

    • The node that handled the request

  3. partition = ...

  • The id of the partition that handled this request

  1. result = :nil

  • The operations' result (nil since the key has no value yet)

Let's set :k1 to 42:

iex(dev@127.0.0.1)2> Civile.Service.put(:k1, 42)
{:ok, :"dev@127.0.0.1",
 913438523331814323877303020447676887284957839360, nil}

We get a tuple as response with 4 items:

  1. status = :ok

    • The request was handled correctly

  2. node = :"dev@127.0.0.1"

    • The node that handled the request

  3. partition = ...

  • The id of the partition that handled this request (the same as before)

  1. result = :nil

  • Nothing of interest to result, use a 3 item tuple to keep the same shape as get

Now let's get :k1 again:

iex(dev@127.0.0.1)3> Civile.Service.get(:k1)
{:ok, :"dev@127.0.0.1",
 913438523331814323877303020447676887284957839360, 42}

Now we get 42 as result

Let's try with another key (:k2):

iex(dev@127.0.0.1)4> Civile.Service.get(:k2)
{:ok, :"dev@127.0.0.1",
 365375409332725729550921208179070754913983135744, nil}

Same response as before, but notice that the partition changed, we will see why later.

Let's set :k2 to :hello:

iex(dev@127.0.0.1)5> Civile.Service.put(:k2, :hello)
{:ok, :"dev@127.0.0.1",
 365375409332725729550921208179070754913983135744, nil}

Now get it:

iex(dev@127.0.0.1)6> Civile.Service.get(:k2)
{:ok, :"dev@127.0.0.1",
 365375409332725729550921208179070754913983135744, :hello}

Now let's build this, first let's create the API on lib/civile_service.ex:

defmodule Civile.Service do
  def ping(v \\ 1) do
    send_cmd("ping#{v}", {:ping, v})
  end

  def put(k, v) do
    send_cmd(k, {:put, {k, v}})
  end

  def get(k) do
    send_cmd(k, {:get, k})
  end

  defp send_cmd(k, cmd) do
    idx = :riak_core_util.chash_key({"civile", k})
    pref_list = :riak_core_apl.get_primary_apl(idx, 1, Civile.Service)

    [{index_node, _type}] = pref_list

    :riak_core_vnode_master.sync_command(index_node, cmd, Civile.VNode_master)
  end
end

The module was refactored to reuse the hashing logic in a common function called send_cmd/2. ping/1, put/2 and get/1 use send_cmd/2 by passing the value that will be used to hash as first argument and the command to send to the vnode as second argument.

On the vnode side we are going to use ets as the storage for our key value store, we need to initialize it in the Civile.VNode.init/1 function:

def init([partition]) do
  table_name = :erlang.list_to_atom('civile_' ++ :erlang.integer_to_list(partition))

  table_id =
    :ets.new(table_name, [:set, {:write_concurrency, false}, {:read_concurrency, false}])

  state = %{
    partition: partition,
    table_name: table_name,
    table_id: table_id
  }

  {:ok, state}
end

We create a table name unique for our app and partition, then create a new ets table that has read and write concurrency set to false since only this vnode will be reading and writing on it.

Now let's handle the put command:

def handle_command({:put, {k, v}}, _sender, state = %{table_id: table_id, partition: partition}) do
  :ets.insert(table_id, {k, v})
  res = {:ok, node(), partition, nil}
  {:reply, res, state}
end

Most of the code is extracting the values we need from the state to run the useful line:

:ets.insert(table_id, {k, v})

And then building the response to return.

The get command:

def handle_command({:get, k}, _sender, state = %{table_id: table_id, partition: partition}) do
  res =
    case :ets.lookup(table_id, k) do
      [] ->
        {:ok, node(), partition, nil}

      [{_, value}] ->
        {:ok, node(), partition, value}
    end

  {:reply, res, state}
end

It calls :ets.lookup/2 and handles the case where there's no value and when one is available.

The project at this stage is available in the tag kv-1

See if the code is correct by compiling the project:

mix compile

And then start one or 3 nodes and try the commands from the top of this chapter, let's do it with 3, first we clean the current state to start fresh:

rm -rf data_* data ring_data_dir*

Now let's run the 3 nodes by setting the right profile on MIX_ENV before running our commands:

MIX_ENV=dev1 iex --name dev1@127.0.0.1 -S mix run
MIX_ENV=dev2 iex --name dev2@127.0.0.1 -S mix run
MIX_ENV=dev3 iex --name dev3@127.0.0.1 -S mix run

On dev2 and dev3 console run the following to join the nodes:

:riak_core.join('dev1@127.0.0.1')

Now let's check the ring status until it settles since we are not migrating data on vnode migration yet:

{:ok, ring} = :riak_core_ring_manager.get_my_ring
:riak_core_ring.pretty_print(ring, [:legend])

From node2:

iex(dev2@127.0.0.1)3> Civile.Service.get(:k1)
{:ok, :"dev1@127.0.0.1",
 913438523331814323877303020447676887284957839360, nil}

From node3:

iex(dev3@127.0.0.1)12> Civile.Service.put(:k1, 42)
{:ok, :"dev1@127.0.0.1",
 913438523331814323877303020447676887284957839360, nil}

From node2:

iex(dev2@127.0.0.1)4> Civile.Service.get(:k1)
{:ok, :"dev1@127.0.0.1",
 913438523331814323877303020447676887284957839360, 42}

We can see that all requests where handled by node1 and the same partition since we used the same key on each request, which was used to decide which vnode/partion would handle it.

Riak Core on Partisan on Elixir Tutorial: Getting Started

Previous posts: Riak Core on Partisan on Elixir Tutorial: Introduction and Riak Core on Partisan on Elixir Tutorial: Setup.

Creating the project

mix new civiledb --app civile
cd civiledb

The project at this stage is available in the tag start-1

Add uniendo dependency

uniendo is a library that contains all required dependencies and overrides in a single place to make it easy to use from other projects, add it to the deps sections in mix.exs like this:

defp deps do
  [
    {:uniendo, git: "git://github.com/marianoguerra/uniendo.git", branch: "master"}
  ]
end

The project at this stage is available in the tag start-2

Add application configuration

We need to add riak_core to the list of applications to start and start our own supervisor tree, modify the application section in mix.exs like this:

def application do
  [
    extra_applications: [:riak_core, :logger],
    mod: {Civile, []}
  ]
end

The project at this stage is available in the tag start-3

First build

Let's try building the project as is, mainly to test that the dependencies are compiled correctly.

First we need to get the dependencies:

mix deps.get

If it's the first time it may ask you:

Could not find Hex, which is needed to build dependency :folsom
Shall I install Hex? (if running non-interactively, use "mix local.hex --force") [Yn]

Just hit enter to continue.

When it finishes, try to build the project:

mix compile

The project at this stage is available in the tag start-4

If it's the first time you build a project with dependencies written in erlang it may ask you:

Shall I install rebar3? (if running non-interactively, use "mix local.rebar --force") [Yn]

Just hit enter to continue.

If you get an error compiling you may want to try updating the rebar3 version mix uses with:

mix local.rebar rebar3 ~/bin/rebar3

If setup fails to build, try commenting the line on deps/setup/rebar.config:

{post_hooks, [{compile, "make escriptize"}]}.

Like this:

%{post_hooks, [{compile, "make escriptize"}]}.

Making it start

We need to start our application and riak_core, for that we need some initial setup:

mkdir priv config
cp deps/riak_core/priv/riak_core.schema priv/

The project at this stage is available in the tag start-5

Add the following content to the file in config/config.exs:

use Mix.Config

config :riak_core,
  ring_state_dir: 'ring_data_dir',
  handoff_port: 8099,
  handoff_ip: '127.0.0.1',
  schema_dirs: ['priv']

config :sasl,
  errlog_type: :error

The project at this stage is available in the tag start-6

Edit lib/civile.ex and change its content to:

defmodule Civile do
  use Application
  require Logger

  def start(_type, _args) do
    case Civile.Supervisor.start_link do
      {:ok, pid} ->
        {:ok, pid}
      {:error, reason} ->
        Logger.error("Unable to start Civile supervisor because: #{inspect reason}")
    end
  end
end

Create a new file called lib/civile_supervisor.ex with the following content:

defmodule Civile.Supervisor do
  use Supervisor

  def start_link do
    # riak_core appends _sup to the application name.
    Supervisor.start_link(__MODULE__, [], [name: :civile_sup])
  end

  def init(_args) do
    children = []
    supervise(children, strategy: :one_for_one, max_restarts: 5, max_seconds: 10)
  end

end

The project at this stage is available in the tag start-7

Now recompile the project:

mix compile

And start it:

iex --name dev@127.0.0.1 -S mix run

You should see something like this:

Erlang/OTP 22 [erts-10.5] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]

00:48:07.530 [info] Starting reporters with []
00:48:07.577 [info] Using node name: 'dev@127.0.0.1'
00:48:07.581 [info] Resolving "127.0.0.1"...
00:48:07.583 [info] Resolved "dev@127.0.0.1" to {127,0,0,1}
00:48:07.583 [info] Resolved "127.0.0.1" to {127,0,0,1}
00:48:07.679 [info] Partisan listening on {127,0,0,1}:49489 listen_addrs: [#{ip => {127,0,0,1},port => 49489}]
00:48:07.686 [info] Not using container orchestration; disabling.
00:48:07.694 [info] node 'dev@127.0.0.1' choosing random seed: {52969312,-576460751045187665,-576460752303423453}
00:48:07.728 [info] node 'dev@127.0.0.1' choosing random seed: {52969312,-576460751045187665,-576460752303423453}
00:48:07.774 [info] Configuring partisan dispatch: false
dets: file "data/cluster_meta/manifest.dets" not properly closed, repairing ...
00:48:07.938 [info] New capability: {riak_core,vnode_routing} = proxy
00:48:07.942 [info] New capability: {riak_core,staged_joins} = true
00:48:07.947 [info] New capability: {riak_core,resizable_ring} = true
00:48:07.951 [info] New capability: {riak_core,fold_req_version} = v2
00:48:07.956 [info] New capability: {riak_core,security} = true
00:48:07.960 [info] New capability: {riak_core,bucket_types} = true
00:48:07.964 [info] New capability: {riak_core,net_ticktime} = true
Interactive Elixir (1.9.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(dev@127.0.0.1)1>

You may get a lager crash log at startup, ignore it.

Hit Ctrl+C twice to quit.

Implementing our VNode

Write the following on a new file called lib/civile_vnode.ex:

defmodule Civile.VNode do
  @behaviour :riak_core_vnode

  def start_vnode(partition) do
    :riak_core_vnode_master.get_vnode_pid(partition, __MODULE__)
  end

  def init([partition]) do
    {:ok, %{partition: partition}}
  end

  def handle_command({:ping, v}, _sender, state = %{partition: partition}) do
     {:reply, {:pong, v + 1, node(), partition}, state}
  end

  def handoff_starting(_dest, state) do
    {true, state}
  end

  def handoff_cancelled(state) do
    {:ok, state}
  end

  def handoff_finished(_dest, state) do
    {:ok, state}
  end

  def handle_handoff_command(_fold_req, _sender, state) do
    {:noreply, state}
  end

  def is_empty(state) do
    {true, state}
  end

  def terminate(_reason, _state) do
    :ok
  end

  def delete(state) do
    {:ok, state}
  end

  def handle_handoff_data(_bin_data, state) do
    {:reply, :ok, state}
  end

  def encode_handoff_item(_k, _v) do
  end

  def handle_coverage(_req, _key_spaces, _sender, state) do
    {:stop, :not_implemented, state}
  end

  def handle_exit(_pid, _reason, state) do
    {:noreply, state}
  end

  def handle_overload_command(_, _, _) do
    :ok
  end

  def handle_overload_info(_, _idx) do
    :ok
  end
end

Write the following on a new file called lib/civile_service.ex:

defmodule Civile.Service do

  def ping(v\\1) do
    idx = :riak_core_util.chash_key({"civile", "ping#{v}"})
    pref_list = :riak_core_apl.get_primary_apl(idx, 1, Civile.Service)

    [{index_node, _type}] = pref_list

    :riak_core_vnode_master.sync_command(index_node, {:ping, v}, Civile.VNode_master)
  end

end

In lib/civile_supervisor.ex, add the vnode master as a child to our supervisor

Change:

def init(_args) do
  children = []
  supervise(children, strategy: :one_for_one, max_restarts: 5, max_seconds: 10)
end

To:

def init(_args) do
  children = [
    worker(:riak_core_vnode_master, [Civile.VNode], id: Civile.VNode_master_worker)
  ]
  supervise(children, strategy: :one_for_one, max_restarts: 5, max_seconds: 10)
end

And register our vnode implementation on riak_core.

In lib/civile.ex change:

def start(_type, _args) do
  case Civile.Supervisor.start_link do
    {:ok, pid} ->
      {:ok, pid}
    {:error, reason} ->
      Logger.error("Unable to start Civile supervisor because: #{inspect reason}")
  end
end

To:

def start(_type, _args) do
  case Civile.Supervisor.start_link do
    {:ok, pid} ->
      :ok = :riak_core.register(vnode_module: Civile.VNode)
      :ok = :riak_core_node_watcher.service_up(Civile.Service, self())
      {:ok, pid}
    {:error, reason} ->
      Logger.error("Unable to start Civile supervisor because: #{inspect reason}")
  end
end

The project at this stage is available in the tag start-8

Now compile and run again:

mix compile
iex --name dev@127.0.0.1 -S mix run

Inside the shell try our new service:

iex(dev@127.0.0.1)1> Civile.Service.ping
{:pong, 2, :"dev@127.0.0.1",
 251195593916248939066258330623111144003363405824}

iex(dev@127.0.0.1)2> Civile.Service.ping 32
{:pong, 33, :"dev@127.0.0.1",
 342539446249430371453988632667878832731859189760}

The response is a tuple that contains the atom :pong, the number we passed (or 1 by default) incremented by one, the node and the partition that handled the reply. Right now we only have one node so that's not useful, but in the next steps it will make more sense.

Playing with clustering (in the same machine)

To build a cluster on the same machine we will need to make 3 builds with slightly different configurations to avoid each running node from reading another node's files or trying to use another node's ports, for this we will create 3 different configs and make 3 different builds.

First add a line at the end of config/config.exs:

import_config "#{Mix.env}.exs"

This will import the configuration for the given mix environment

Now create the following files:

config/dev1.exs

use Mix.Config

config :riak_core,
  node: 'dev1@127.0.0.1',
  web_port: 8198,
  handoff_port: 8199,
  ring_state_dir: 'ring_data_dir_1',
  platform_data_dir: 'data_1'

config/dev2.exs

use Mix.Config

config :riak_core,
  node: 'dev2@127.0.0.1',
  web_port: 8298,
  handoff_port: 8299,
  ring_state_dir: 'ring_data_dir_2',
  platform_data_dir: 'data_2'

config/dev3.exs

use Mix.Config

config :riak_core,
  node: 'dev3@127.0.0.1',
  web_port: 8398,
  handoff_port: 8399,
  ring_state_dir: 'ring_data_dir_3',
  platform_data_dir: 'data_3'

Since the default environment is dev we need to have a file for that one too, but without any config since the one in config.exs is ok:

config/dev.exs

use Mix.Config

The project at this stage is available in the tag start-9

Now let's run the 3 nodes by setting the right profile on MIX_ENV before running our commands:

MIX_ENV=dev1 iex --name dev1@127.0.0.1 -S mix run
MIX_ENV=dev2 iex --name dev2@127.0.0.1 -S mix run
MIX_ENV=dev3 iex --name dev3@127.0.0.1 -S mix run

On dev2 and dev3 console run the following to join the nodes:

:riak_core.join('dev1@127.0.0.1')

Now let's check the ring status:

{:ok, ring} = :riak_core_ring_manager.get_my_ring
:riak_core_ring.pretty_print(ring, [:legend])

It will take a while until the ring rebalances, run the two lines above periodically until they settle.

Now you can run ping from any node and it will work:

Civile.Service.ping 32

Given that the number we pass is used to decide the target vnode, this means that a call with the same number will end up on the same node, but from all nodes it will work transparently.

Running it on the 3 nodes after the cluster rebalanced at around 33% each I got:

iex(dev1@127.0.0.1)2> Civile.Service.ping 32
{:pong, 33, :"dev3@127.0.0.1",
 342539446249430371453988632667878832731859189760}
iex(dev2@127.0.0.1)5> Civile.Service.ping 32
{:pong, 33, :"dev3@127.0.0.1",
 342539446249430371453988632667878832731859189760}
iex(dev3@127.0.0.1)12> Civile.Service.ping 32
{:pong, 33, :"dev3@127.0.0.1",
 342539446249430371453988632667878832731859189760}

This means that the same value was handled by the same node and vnode no matter where the function was called.