A rebar3 Cowboy REST app and a template

1   Introduction

This is another attempt to explain how to implement HTTP REST applications. This one uses the REST support of the Erlang Cowboy Web server (https://ninenines.eu/).

We also look at a template for the REST handler file.

1.1   More information and help

The framework for REST handlers used by Cowboy and Webmachine (and other Erlang Web frameworks?) are very similar. So, the documentation on Webmachine may be helpful with Cowboy REST, also. You can find help with Webmachine here: https://en.wikiversity.org/wiki/Web_Development_with_Webmachine_for_Erlang

1.2   Supporting files

The files of interest for this discussion are in the following archive: test07a_source.zip

This archive contains:

  • rebar.config -- A rebar3 configuration file.

-config/sys.config -- A configuration file. You can place values that you want globally available in your application here. See Additional configuration for more information.

  • config/vm.args

  • vim_cowboy_rest_template -- A template or skeleton file for the Vim text editor. (see below)

  • apps/test07a/src/test07a.app.src -- A configuration file for this application. I've added cowboy, sha3, hex (used by sha3), and jiffy (JSON support) as dependencies.

  • vim_cowboy_rest_template -- This template can be copied into your src/ directory and renamed as your <my_handler>.erl REST handler module, or, more conveniently, it is intended be inserted into a file with the vim text editor. You can also find this file in a fork of the Vim Erlang Skeletons Github repository here: https://github.com/dkuhlman/vim-erlang-skeletons. And, the original repository from which this one was forked, is here: https://github.com/vim-erlang/vim-erlang-skeletons. In order to use it in Vim, you will need to install it in your .vim directory. For example, if you use the pathogen package manager for Vim, you can do:

    $ cd ~/.vim/bundle
    $ git clone https://github.com/dkuhlman/vim-erlang-skeletons
    

    Also see The Cowboy REST handler template elsewhere in this document.

2   Generating a release skeleton with rebar3

A note on releases and applications -- With rebar3 we can generate the directories and starter files for a "release". By default, when we do so, we get a files for a single application (under the apps/ directory). Afterward, we can generate additional applications. Here's how.

Generate the skeleton of a release with rebar3. And, optionally, generate one additional application (test1a)):

$ rebar3 new release test1
$ cd test1
$ cd apps/
$ rebar3 new app test1a

Notes:

  • rebar3 new release generates a directory contain the release.
  • rebar3 new app generates one additional application in this release. When you "start" the release, i.e. you run it, both of these applications will be started. So, for example, you could have one of these applications listening on one port, and the other listening on another. Or, you could have one of these applications handling (routed to) one set of paths (in the URL), and the other application handling another set.

You will then have a directory structure that looks like this:

test07/
├── apps
│   ├── test07
│   │   └── src
│   └── test07a
│       └── src
└── config

Notes:

  • We will be putting the source code for our application in apps/test07/src or apps/test07/src. Generating both of those gives us the option of implementing two applications that run under the same server. We could, of course, create more with the rebar3 new app command, as shown above.
  • We will also do some configuration in test07/rebar.config.
  • We can define globally accessible values in test07/config/sys.config.

Edit rebar.config with your favorite text editor. And add the additional app test1a and any needed dependencies, in particular, cowboy:

{erl_opts, [debug_info]}.
{deps, [
        {cowboy, {git, "https://github.com/ninenines/cowboy.git"}},
        {jiffy, {git, "https://github.com/davisp/jiffy.git", {branch, "master"}}},
        {sha3, {git, "https://github.com/b/sha3.git", {branch, "master"}}},
        {hex, {git, "git://github.com/b/hex", {branch, "master"}}}
       ]}.

{plugins, [rebar3_run]}.

{relx, [{release, { test07, "0.1.0" },
         [test07,
          test07a,
          sasl]},
        {sys_config, "./config/sys.config"},
        {vm_args, "./config/vm.args"},
        {dev_mode, true},
        {include_erts, false},
        {extended_start_script, true}]
}.

{profiles, [
            {prod, [
                    {relx, [
                            {dev_mode, false},
                            {include_erts, true}]}
                   ]
            }]
}.

Notes:

Compile the applications (rebar3 will automatically download any needed dependencies):

$ rebar3 compile

You can build a release and run it:

$ rebar3 release

And, then, run it with something like the following:

$ _build/default/rel/test07/bin/test07 console

Notes:

  • You will need to change the name of the release.

Or, you can build the release and run it with a single command:

$ rebar3 run

After you have built a release, and as long as you do not add applications or dependencies, you can compile and run more quickly with these commands:

$ rebar3 compile
$ _build/default/rel/test1/bin/test1 console

Or, alternatively, the following will recompile your applications, start the applications, and give you an erl shell interactive prompt:

$ rebar3 shell

2.1   Recompiling and reloading

Note: I had problems with the methods in this section, so you will need to experiment with them yourself.

If you started your release with rebar3 shell, then you can quickly recompile any changed files and reload them without leaving the shell:

> r3:do(compile).

As an alternative, there is also the rebar3_auto plugin. Put this in your ~/.config/rebar3/rebar.config config file:

{plugins, [rebar3_auto]}.

In order for the above to work, you might need to install inotify-tools, which on Linux, you can do with apt-get or aptitude.

More information is here: https://www.rebar3.org/docs/using-available-plugins#auto-compile-and-load

2.2   A note on dependencies

You might want to consider doing:

$ rebar3 update

After doing so, you might be able to refer to dependencies, in rebar.config, more briefly. For example:

{deps, [
        cowboy,
        jiffy,
        sha3
       ]}.

I say "might be able to", because I'm not sure about which version this will download. I suspect that it is the latest version referred to in the rebar3 package index. You can display that package index by running the following:

$ rebar3 pkgs

2.3   Additional configuration

You can specify globally available values in config/sys.config. For example, in the example application we are discussing, we need the name and location of a DETS data file:

[
 { test07, [
        {data_file_name,
         "/path/to/directory/containing/test07/Data/test07_data01.dets"}
       ]},
 { test07a, [
        {data_file_name,
         "/path/to/directory/containing/test07/Data/test07a_data01.dets"}
       ]}
].

Then you can obtain the value associated with a name with something like the following:

{ok, DataFileName} = application:get_env(data_file_name),

You can find more information about configuration here: http://erlang.org/doc/man/config.html, and here: http://erlang.org/doc/design_principles/applications.html.

If you generated an additional application, then you may need to do some configuration in the .app.src file for that application. For example, in apps/test07a/src/test07a.app.src, I added additional applications:

{application, test07a,
 [{description, "An OTP application"},
  {vsn, "0.1.0"},
  {registered, []},
  {mod, { test07a_app, []}},
  {applications,
   [kernel,
    stdlib,
    cowboy,
    sha3,
    hex,
    jiffy
   ]},
  {env,[]},
  {modules, []},

  {maintainers, []},
  {licenses, []},
  {links, []}
 ]}.

3   Implementing the REST modules

Basically, you need to do two things: (1) implement a *_app.erl module and (2) implement a handler.

3.1   The application start-up

In the app module (test07/apps/test07a/src/test07a_app.erl, in our example), we need to do the following:

  • Define the routes, i.e. the paths that the client will place on any request URL along with the module that will handle the request with that path.
  • Start any additional processes -- In this example, we start a data base server that has been implemented with the gen_server behavior.
  • Start the supervisor for application itself.

Here is an example of that code:

-module(test07a_app).
-behaviour(application).
-export([start/2, stop/1]).

start(_Type, _Args) ->
    Dispatch = cowboy_router:compile([
        {'_',
         [
          {"/docs/get/:doc_id", test07a_handler, [{op, get}]},
          {"/docs/list", test07a_handler, [{op, list}]},
          {"/docs/create", test07a_handler, [{op, create}]},
          {"/docs/update/:doc_id", test07a_handler, [{op, update}]},
          {"/docs/delete/:doc_id", test07a_handler, [{op, delete}]},
          {"/docs/help", test07a_handler, [{op, help}]},
          {"/docs", test07a_handler, [{op, help}]}
         ]}
                                     ]),
    {ok, _} = cowboy:start_clear(test07a_http_listener, 100,
                                 [{port, 8002}],
                                 #{env => #{dispatch => Dispatch}}
                                ),
    db_sup:start_link(),
    test07a_sup:start_link().

stop(_State) ->
    ok.

Notes:

  • The above routes the URL/path http://somehost:8002/list to the handler module test07a_handler.erl, and passes the options [{op, list}] to the init/2 function.

  • In a similar way, it routes the path /docs/get/some_id, to module test07a_handler.erl where we can capture the doc_id, with the following:

    RecordId = cowboy_req:binding(doc_id, Req),
    
  • We can capture those options in function init/2, and pass them to other callback functions called during request processing with the following:

    -record(state, {op, response}).
    
    init(Req, Opts) ->
        [{op, Op} | _] = Opts,
        State = #state{op=Op, response=none},
        {cowboy_rest, Req, State}.
    

3.2   The REST handler

In the REST handler module (apps/test07a/src/test07a_handler.erl in our example) we need to (1) tell cowboy that this is a REST module and (2) implement various callbacks needed to handle requests in the way we need them handled.

We handle the first requirement by returning cowboy_rest from our init function:

-record(state, {op, response}).

init(Req, Opts) ->
    [{op, Op} | _] = Opts,
    State = #state{op=Op, response=none},
    {cowboy_rest, Req, State}.

We handle the second requirement by (1) initializing our handler module with the handler template (vim_cowboy_rest_template) and then (2) uncommenting and implementing any callbacks whose default implementations do not fit our needs, and (3) add additional callback functions for each content type you intend to deliver (e.g. in response to GET requests) and each content type you intend to accept (e.g. in response to PUT and POST requests)..

3.2.1   The Cowboy REST handler template

The Cowboy REST handler template can be obtained from my fork of the vim-erlang-skeletons repository at Github: https://github.com/dkuhlman/vim-erlang-skeletons.

If you use Vim as your text editor, then do the following:

$ cd ~/.vim/bundle
$ git clone https://github.com/dkuhlman/vim-erlang-skeletons.git
$ cd /path/to/myapp
$ vim apps/myapp/src/myapp_handler.erl

And, then, inside Vim, edit an empty .erl file and do: :ErlCowboyREST.

If you do not use Vim, then you can find the template in the archive (test07a_source.zip); it's in the file named vim_cowboy_rest_template. Copy it and then edit "$author", "$year", etc.

3.2.2   Uncommenting and implementing callbacks

It's a bit beyond the scope of this post to explain the details of implementing each of Cowboy's REST callback functions. However, you can find help here:

You can look at the commented-out code in the template to determine the default return values.

And, here are a few hints and suggestions on which callbacks to implement:

  • You can look at the commented-out code in the template to determine the default return values.

  • allowed_methods and known_methods -- Any methods that you want your application to respond to must be in the lists returned by these methods. If a client makes a request with a method that is not in this list, then none of your code will be called and the client will receive a response:

    HTTP/1.1 501 Not Implemented
    

    Or:

    HTTP/1.1 405 Method Not Allowed
    allow: GET, POST, DELETE
    

    So, add or remove methods from the default to fit your needs. See the Cowboy REST flowcharts for additional information: https://ninenines.eu/docs/en/cowboy/2.0/guide/rest_flowcharts/.

  • content_types_provided -- If you want to deliver content types other than text/html, implement this one and specify the function that should be called to produce that content.

  • content_types_accepted -- This one pertains to POST and PUT methods in particular. The HTTP client will specify the content type of the data submitted with the request. Uncomment and implement this callback so that it specifies what content types your application will respond to and the function that Cowboy should call to produce that response.

  • delete_resource and delete_completed -- If your application needs to respond to an HTTP DELETE method and needs to delete a resource, then implement delete_resource and possibly delete_completed. You might also consider implementing the resource_exists callback, which will determine whether the client receives a "404 Not Found" response.

  • resource_exists -- If you want to terminate processing of the request depending on whether or not a resource exists, implement this callback. See the REST flow charts to determine the logic controlled by whether resource_exists returns true or false: https://ninenines.eu/docs/en/cowboy/2.0/guide/rest_flowcharts/.

  • The questions and answers in the following page are very helpful: https://ninenines.eu/docs/en/cowboy/2.0/guide/resource_design/.

3.2.3   Add additional callbacks

If you specified any content types that your handler will accept or provide, then you must implement the callback functions that you specified for each content type. For example, given the following implementations of these callbacks functions:

content_types_accepted(Req, State) ->
    Value = [
             {<<"application/x-www-form-urlencoded">>, from_json}
            ],
    {Value, Req, State}.

content_types_provided(Req, State) ->
    Value = [
             {{ <<"application">>, <<"json">>, '*'}, to_json}
            ],
    {Value, Req, State}.

Then you will need to implement callback functions from_json/2 and to_json/2.

Here are a few hints about implementing these additional callback functions:

  • For content_types_accepted callback functions, return {{true, Content}, Req, State}.

  • For content_types_provided callback functions, return ``{Content, Req, State}.

  • If the route/path specifies a variable, then you can obtain the value associated with that variable with the following:

    VariableValue = cowboy_req:binding(variable_name, Req),
    
  • If the request includes form data (perhaps URL encoded data specified using curl --data-urlencode <data>), then you can obtain the value associated with a form variable (in the following case the name "content") with the something like the following:

    {ok, ValueList, Req1}= cowboy_req:read_urlencoded_body(Req),
    Content = proplists:get_value(<<"content">>, ValueList),
    
  • In my application, I'm returning JSON content. So, I produce that JSON content by constructing a suitable Erlang term, then converting it to a JSON object or dictionary with jiffy:encode/1. For example:

    Data = {[{<<"key1">>, <<"value 1">>}, {<<"key2">>, <<"value 2">>}]},
    Body = jiffy:encode(Data),
    {Body, Req, State}.
    

    Note that there are other JSON encoders/decoders for Erlang. I found jiffy to be convenient, but you might also want to look at:

    • mochijson2 -- mochijson2 and mochijson are part of the mochiweb project.
    • JSX
  • In order to delete a resource, implement the delete_resource/2 callback function. It will be called in response to a request that uses the HTTP DELETE method. In the delete_resource/2 callback, you should (1) delete the resource and (2) return {true, Req, State} if your delete action succeeded, but return {false, Req, State} if it failed.

links