Dave's page

A Cowboy REST application with add, get, update, and list

1   Introduction

This Erlang project can be viewed as another example of how to do REST on top of the Cowboy Web server. For other Cowboy REST examples, see the examples subdirectory for Cowboy: https://github.com/ninenines/cowboy

This example Cowboy REST application is a CRUD application: it implements and makes available an HTTP API that does create, read, update, and delete operations on resources. For more about CRUD, see: https://en.wikipedia.org/wiki/Create,_read,_update_and_delete.

There is a repository containing this application here: https://github.com/dkuhlman/cowboy_rest_crud.

2   Creating the application

I followed the recipe from the Cowboy "User guide" to create the initial application. See the instructions here: https://ninenines.eu/docs/en/cowboy/2.0/guide/getting_started/.

3   Creating the database

The following Erlang script will create and initialize a new database that can be used by this application:

#!/usr/bin/env escript

%%
%% create_table.escript
%%
%% synopsis:
%%     Create new records and state DETS files.
%% usage:
%%     create_table.escript <db-file-stem>
%% example:
%%     # create Data/test01_records.dets and Data/test01_state.dets
%%     create_table.escript Data/test01
%%

main([Filename]) ->
    Filename1 = io_lib:format("~s_records.dets", [Filename]),
    Filename2 = io_lib:format("~s_state.dets", [Filename]),
    dets:open_file(record_tab, [{file, Filename1}, {type, set}]),
    dets:open_file(record_state_tab, [{file, Filename2}, {type, set}]),
    dets:insert(record_state_tab, {current_id, 1}),
    dets:close(record_state_tab),
    dets:close(record_tab);
main(["--help"]) ->
    usage();
main([]) ->
    usage().

usage() ->
    io:fwrite("usage: create_table.escript <db-file-stem>~n", []).

You will need to add the location of the resulting two files to your rel/sys.config. See section The configuration file for more on that.

4   Client use of the app

4.1   Using curl

Here are the commands that I used to test this example REST application. You can find documentation on cUrl here: https://curl.haxx.se/. If you cloned the repository (at https://github.com/dkuhlman/cowboy_rest_crud), then the bin subdirectory will contain shell scripts for these commands.

Add a record -- This curl script creates a record from a text file:

#!/bin/bash
curl -v --data-urlencode content@$1 http://crow.local:8080/create

List the existing records; get JSON:

#!/bin/bash
curl -H "Accept: application/json" http://crow.local:8080/list
echo

List the existing records; get plain text:

#!/bin/bash
curl -H "Accept: text/plain" http://crow.local:8080/list
echo

Update a record, replacing contents with data from a local file:

#!/bin/bash
curl -v --data-urlencode content@$2 http://crow.local:8080/update/$1

Get/retrieve a specific record by ID; return JSON:

#!/bin/bash
curl -H "Accept: application/json" http://crow.local:8080/get/$1
echo

Get/retrieve a specific record by ID; return plain text:

#!/bin/bash
curl -H "Accept: text/plain" http://crow.local:8080/get/$1
echo

Delete a specific record by its ID:

#!/bin/bash
curl -X "DELETE" http://crow.local:8080/delete/$1
echo

Get the help message; return JSON:

#!/bin/bash
curl -H "Accept: application/json" http://crow.local:8080/help
echo

Get the help message; return plain text:

#!/bin/bash
curl -H "Accept: application/json" http://crow.local:8080/help
echo

Notes:

  • Because this application uses content_types_provided/2 to deliver several content types when the client requests a record, a list of records, or the help message, we need to specify the content type with -H "Accept: xxxx" in the cUrl request.

5   Guidance and explanations

5.1   init/2

You will want to implement the init/2 callback function for several reasons. First, by returning the atom cowboy_rest, you tell Cowboy to follow its REST logic. And, second, init/2 gives you a way to capture options that are specific to a particular routing URL, in effect giving you a way to specify (static) options for each item in your REST API.

As its second argument, the init/2 callback function in the handler takes the options from the URL path specified in your routings, in this example, that's in src/rest_update_app.erl. In order to pass it along to other callbacks, you may want to define and use a state record. For example:

-record(state, {op}).

init(Req, Opts) ->
    [Op | _] = Opts,
    State = #state{op=Op},
    {cowboy_rest, Req, State}.

5.2   Get a resource

In order to handle an HTTP GET method, do the following:

  1. Add <<"GET">> to the return values of allowed_methods/2. Example:

    allowed_methods(Req, State) ->
    Methods = [<<"GET">>, <<"POST">>, <<"DELETE">>],
    {Methods, Req, State}.

    Note that if you do not implement the allowed_methods/2 callback in your handler, the default value is [<<"GET">>, <<"HEAD">>, <<"OPTIONS">>]. So, it is possible that you will not need to implement this callback. See: https://ninenines.eu/docs/en/cowboy/2.0/guide/rest_handlers/

  2. For each different type of content that you want your clients to be able to request and that you want to return to clients, add an entry to the return value of the content_types_provided/2 callback function specifying the content type and the function that produces it. Example:

    content_types_provided(Req, State) ->
    {[
      {<<"application/json">>, db_to_json}
     ], Req, State}.

  3. Implement a callback function that produces that content type. Example:

    db_to_json(Req, #state{op=Op} = State) ->
    {Body, Req1, State1} = case Op of
        list ->
            get_record_list(Req, State);
        get ->
            get_one_record(Req, State);
        help ->
            get_help(Req, State)
    end,
    {Body, Req1, State1}.

get_one_record(Req, State) ->
    RecordId = cowboy_req:binding(record_id, Req),
    RecordId1 = binary_to_list(RecordId),
    {ok, Recordfilename} = application:get_env(rest_update, records_file_name),
    {ok, _} = dets:open_file(records_db, [{file, Recordfilename}, {type, set}]),
    Records = dets:lookup(records_db, RecordId1),
    ok = dets:close(records_db),
    Body = case Records of
        [{RecordId2, Data}] ->
            io_lib:format("{\"id\": \"~s\", \"record\": \"~s\"}",
                          [RecordId2, binary_to_list(Data)]);
        [] ->
            io_lib:format("{\"not_found\": \"record ~p not found\"}",
                          [RecordId1]);
        _ ->
            io_lib:format("{\"extra_records\": \"extra records for ~p\"}",
                          [RecordId1])
    end,
    {list_to_binary(Body), Req, State}.

    Note that the return value of this function is JSON text that has been converted to an Erlang binary.

5.3   Delete a resource

In order to handle an HTTP DELETE method you must do the following:

  1. Add <<"DELETE">> to the return values of allowed_methods/2. Example:

    allowed_methods(Req, State) ->
    Methods = [<<"GET">>, <<"POST">>, <<"DELETE">>],
    {Methods, Req, State}.

  2. Implement the delete_resource/2 callback function, which should actually delete or remove the resource. Example:

    delete_resource(Req, State) ->
    io:fwrite("(delete_resource) testing.~n", []),
    RecordId = cowboy_req:binding(record_id, Req),
    RecordId1 = binary_to_list(RecordId),
    {ok, Recordfilename} = application:get_env(rest_update, records_file_name),
    {ok, _} = dets:open_file(records_db, [{file, Recordfilename}, {type, set}]),
    Result = dets:delete(records_db, RecordId1),
    ok = dets:close(records_db),
    Response = case Result of
        ok ->
            true;
        {error, _Reason} ->
            false
    end,
    {Response, Req, State}.

  3. Optionally, you can implement the resource_exists/2 callback function, which should return true if you want Cowboy to call delete_resource/2 and false if not. You can look at the Cowboy REST flowcharts to determine which other callback functions are called or not depending on the value returned by resource_exists/2. See: https://ninenines.eu/docs/en/cowboy/2.0/guide/rest_flowcharts/

5.4   Update a resource

Here is the code that does our update:

update_record_to_json(Req, State) ->
    case cowboy_req:method(Req) of
        <<"POST">> ->
            RecId = cowboy_req:binding(record_id, Req),
            RecId1 = binary_to_list(RecId),
            {ok, [{<<"content">>, NewContent}], Req1} =
                cowboy_req:read_urlencoded_body(Req),
            {ok, Recordfilename} = application:get_env(
                rest_update, records_file_name),
            {ok, _} = dets:open_file(
                records_db, [{file, Recordfilename}, {type, set}]),
            DBResponse = dets:lookup(records_db, RecId1),
            Result = case DBResponse of
                [_] ->
                    ok = dets:insert(records_db, {RecId1, NewContent}),
                    ok = dets:sync(records_db),
                    Response = io_lib:format("/get/~s", [RecId1]),
                    Response1 = list_to_binary(Response),
                    {{true, Response1}, Req1, State};
                [] ->
                    {true, Req1, State}
            end,
            ok = dets:close(records_db),
            Result;
        _ ->
            {true, Req, State}
    end.

Notes:

  • We only want to do this when we get a POST method. I'm not sure that the check for this is needed, however.

6   The code

6.1   The make file

Below is most of the code for this project. The complete project can be found here: https://github.com/dkuhlman/cowboy_rest_crud.

Makefile:

PROJECT = rest_update
PROJECT_DESCRIPTION = A Cowboy REST update DETS project
PROJECT_VERSION = 0.1.0

DEPS = cowboy
dep_cowboy_commit = master

include erlang.mk

Notes:

  • We've added Cowboy as a dependency.

6.2   The configuration file

rel/sys.config:

[
    {rest_update,
        [
            {records_file_name, "/home/dkuhlman/a1/Erlang/Cowboy/Work/rest_update/Data/records01_records.dets"},
            {state_file_name, "/home/dkuhlman/a1/Erlang/Cowboy/Work/rest_update/Data/records01_state.dets"}
        ]
    }
].

Notes:

  • At runtime, we will need the path to and name of the two DETS files that hold the data and the current/next ID index (used to create a unique ID for each new record). In our handler, we can retrieve this information with something like the following:

    {ok, Recordfilename} = application:get_env(rest_update, records_file_name),
{ok, Statefilename} = application:get_env(rest_update, state_file_name),

6.3   The supervisor

src/rest_update_sup.erl:

-module(rest_update_sup).
-behaviour(supervisor).

-export([start_link/0]).
-export([init/1]).

start_link() ->
    supervisor:start_link({local, ?MODULE}, ?MODULE, []).

init([]) ->
    Procs = [],
    {ok, {{one_for_one, 1, 5}, Procs}}.

6.4   The app

src/rest_update_app.erl:

-module(rest_update_app).
-behaviour(application).

-export([start/2]).
-export([stop/1]).

start(_Type, _Args) ->
    Dispatch = cowboy_router:compile([
        {'_', [
               {"/list", db_update_handler, [list]},
               {"/get/:record_id", db_update_handler, [get]},
               {"/create", db_update_handler, [create]},
               {"/update/:record_id", db_update_handler, [update]},
               {"/delete/:record_id", db_update_handler, [delete]},
               {"/help", db_update_handler, [help]},
               {"/", db_update_handler, [help]}
              ]}
    ]),
    {ok, _} = cowboy:start_clear(my_http_listener, 100,
        [{port, 8080}],
        #{env => #{dispatch => Dispatch}}
    ),
    rest_update_sup:start_link().

stop(_State) ->
    ok.

Notes:

  • For our purposes, the most important thing we do is to provide routing information for the various URLs that we intend our clients to request.

  • The portion of the path that begins with a colon (":") will be passed in as part of the request so that we can retrieve it. In our handler we can retrieve it by calling cowboy_req:binding/2. Here is an example:

    delete_resource(Req, State) ->
    io:fwrite("(delete_resource) testing.~n", []),
    RecordId = cowboy_req:binding(record_id, Req),

  • The third argument is a value that is passed to our init/2 function of our handler. That value can be any Erlang term: a list, a tuple, an atom, etc. We have passed a list with a single atom in each case.

  • The calls to cowboy:start_clear/4 seems special to Cowboy 2.0, I believe. Make sure that you follow the User Guide for the specific version of Cowboy that you intend to use, that is, either Cowboy 1.0 or 2.0.

6.5   The handler

src/db_update_handler.erl:

-module(db_update_handler).

%% Webmachine API
-export([
         init/2,
         allowed_methods/2,
         content_types_provided/2,
         content_types_accepted/2,
         resource_exists/2,
         delete_resource/2
        ]).

-export([
         db_to_json/2,
         db_to_text/2,
         text_to_db/2
        ]).

-record(state, {op}).

init(Req, Opts) ->
    [Op | _] = Opts,
    State = #state{op=Op},
    {cowboy_rest, Req, State}.

allowed_methods(Req, State) ->
    Methods = [<<"GET">>, <<"POST">>, <<"DELETE">>],
    {Methods, Req, State}.

content_types_provided(Req, State) ->
    {[
      {<<"application/json">>, db_to_json},
      {<<"text/plain">>, db_to_text}
     ], Req, State}.

content_types_accepted(Req, State) ->
    {[
      {<<"text/plain">>, text_to_db},
      %{<<"application/json">>, text_to_db}
      {<<"application/x-www-form-urlencoded">>, text_to_db}
     ], Req, State}.

db_to_json(Req, #state{op=Op} = State) ->
    {Body, Req1, State1} = case Op of
        list ->
            get_record_list(Req, State);
        get ->
            get_one_record(Req, State);
        help ->
            get_help(Req, State)
    end,
    {Body, Req1, State1}.

db_to_text(Req, #state{op=Op} = State) ->
    {Body, Req1, State1} = case Op of
        list ->
            get_record_list_text(Req, State);
        get ->
            get_one_record_text(Req, State);
        help ->
            get_help_text(Req, State)
    end,
    {Body, Req1, State1}.

text_to_db(Req, #state{op=Op} = State) ->
    {Body, Req1, State1} = case Op of
        create ->
            create_record_to_json(Req, State);
        delete ->
            delete_record_to_json(Req, State);
        update ->
            update_record_to_json(Req, State)
    end,
    {Body, Req1, State1}.

resource_exists(Req, State) ->
    case cowboy_req:method(Req) of
        <<"DELETE">> ->
            RecordId = cowboy_req:binding(record_id, Req),
            RecordId1 = binary_to_list(RecordId),
            {ok, Recordfilename} = application:get_env(
                 rest_update, records_file_name),
            {ok, _} = dets:open_file(
                records_db, [{file, Recordfilename}, {type, set}]),
            Records = dets:lookup(records_db, RecordId1),
            ok = dets:close(records_db),
            Response = case Records of
                [_] ->
                    {true, Req, State};
                _ ->
                    {false, Req, State}
            end,
            Response;
        _ ->
            {true, Req, State}
    end.

delete_resource(Req, State) ->
    RecordId = cowboy_req:binding(record_id, Req),
    RecordId1 = binary_to_list(RecordId),
    {ok, Recordfilename} = application:get_env(rest_update, records_file_name),
    {ok, _} = dets:open_file(records_db, [{file, Recordfilename}, {type, set}]),
    Result = dets:delete(records_db, RecordId1),
    ok = dets:close(records_db),
    Response = case Result of
        ok ->
            true;
        {error, _Reason} ->
            false
    end,
    {Response, Req, State}.

get_record_list(Req, State) ->
    {ok, Recordfilename} = application:get_env(rest_update, records_file_name),
    dets:open_file(records_db, [{file, Recordfilename}, {type, set}]),
    %F = fun (Item, Acc) -> Acc1 = [Item | Acc], Acc1 end,
    F = fun (Item, Acc) ->
                {Id, Rec} = Item,
                Rec1 = re:replace(Rec, "\n", "\\\n",
                                  [{return, list}, global]),
                Item1 = io_lib:format("~p: ~p",
                    [Id, Rec1]),
                [lists:flatten(Item1) | Acc]
        end,
    Items = dets:foldl(F, [], records_db),
    dets:close(records_db),
    Items1 = lists:sort(Items),
    Items2 = lists:flatten(lists:join(",\n", Items1)),
    Body = "
{
    \"list\": {~s}
}",
    Body1 = io_lib:format(Body, [Items2]),
    {Body1, Req, State}.

get_record_list_text(Req, State) ->
    {ok, Recordfilename} = application:get_env(rest_update, records_file_name),
    dets:open_file(records_db, [{file, Recordfilename}, {type, set}]),
    F = fun (Item, Acc) -> Acc1 = [Item | Acc], Acc1 end,
    Items = dets:foldl(F, [], records_db),
    dets:close(records_db),
    Items1 = lists:sort(Items),
    Body = "
list: ~p,
",
    Body1 = io_lib:format(Body, [Items1]),
    {Body1, Req, State}.

get_one_record(Req, State) ->
    RecordId = cowboy_req:binding(record_id, Req),
    RecordId1 = binary_to_list(RecordId),
    {ok, Recordfilename} = application:get_env(rest_update, records_file_name),
    {ok, _} = dets:open_file(records_db, [{file, Recordfilename}, {type, set}]),
    Records = dets:lookup(records_db, RecordId1),
    ok = dets:close(records_db),
    Body = case Records of
        [{RecordId2, Data}] ->
            io_lib:format("{\"id\": \"~s\", \"record\": \"~s\"}",
                          [RecordId2, binary_to_list(Data)]);
        [] ->
            io_lib:format("{\"not_found\": \"record ~p not found\"}",
                          [RecordId1]);
        _ ->
            io_lib:format("{\"extra_records\": \"extra records for ~p\"}",
                          [RecordId1])
    end,
    {list_to_binary(Body), Req, State}.

get_one_record_text(Req, State) ->
    RecordId = cowboy_req:binding(record_id, Req),
    RecordId1 = binary_to_list(RecordId),
    {ok, Recordfilename} = application:get_env(rest_update, records_file_name),
    {ok, _} = dets:open_file(records_db, [{file, Recordfilename}, {type, set}]),
    Records = dets:lookup(records_db, RecordId1),
    ok = dets:close(records_db),
    Body = case Records of
        [{RecordId2, Data}] ->
            io_lib:format("id: \"~s\", record: \"~s\"",
                          [RecordId2, binary_to_list(Data)]);
        [] ->
            io_lib:format("{not_found: record ~p not found",
                          [RecordId1]);
        _ ->
            io_lib:format("{extra_records: extra records for ~p",
                          [RecordId1])
    end,
    {list_to_binary(Body), Req, State}.

create_record_to_json(Req, State) ->
    {ok, [{<<"content">>, Content}], Req1} = cowboy_req:read_urlencoded_body(Req),
    RecordId = generate_id(),
    {ok, Recordfilename} = application:get_env(rest_update, records_file_name),
    {ok, _} = dets:open_file(records_db, [{file, Recordfilename}, {type, set}]),
    ok = dets:insert(records_db, {RecordId, Content}),
    ok = dets:sync(records_db),
    ok = dets:close(records_db),
    case cowboy_req:method(Req1) of
        <<"POST">> ->
            Response = io_lib:format("/get/~s", [RecordId]),
            {{true, list_to_binary(Response)}, Req1, State};
        _ ->
            {true, Req1, State}
    end.

update_record_to_json(Req, State) ->
    case cowboy_req:method(Req) of
        <<"POST">> ->
            RecId = cowboy_req:binding(record_id, Req),
            RecId1 = binary_to_list(RecId),
            {ok, [{<<"content">>, NewContent}], Req1} =
                cowboy_req:read_urlencoded_body(Req),
            {ok, Recordfilename} = application:get_env(
                rest_update, records_file_name),
            {ok, _} = dets:open_file(
                records_db, [{file, Recordfilename}, {type, set}]),
            DBResponse = dets:lookup(records_db, RecId1),
            Result = case DBResponse of
                [_] ->
                    ok = dets:insert(records_db, {RecId1, NewContent}),
                    ok = dets:sync(records_db),
                    Response = io_lib:format("/get/~s", [RecId1]),
                    Response1 = list_to_binary(Response),
                    {{true, Response1}, Req1, State};
                [] ->
                    {true, Req1, State}
            end,
            ok = dets:close(records_db),
            Result;
        _ ->
            {true, Req, State}
    end.

delete_record_to_json(Req, State) ->
    case cowboy_req:method(Req) of
        <<"POST">> ->
            RecId = cowboy_req:binding(record_id, Req),
            RecId1 = binary_to_list(RecId),
            {ok, Recordfilename} = application:get_env(
                rest_update, records_file_name),
            {ok, _} = dets:open_file(
                records_db, [{file, Recordfilename}, {type, set}]),
            DBResponse = dets:lookup(records_db, RecId1),
            Result = case DBResponse of
                [_] ->
                    ok = dets:delete(records_db, RecId1),
                    ok = dets:sync(records_db),
                    Response = io_lib:format("/delete/~s", [RecId1]),
                    Response1 = list_to_binary(Response),
                    {{true, Response1}, Req, State};
                [] ->
                    {true, Req, State}
            end,
            ok = dets:close(records_db),
            Result;
        _ ->
            {true, Req, State}
    end.

get_help(Req, State) ->
    {ok, Recordfilename} = application:get_env(rest_update, records_file_name),
    {ok, Statefilename} = application:get_env(rest_update, state_file_name),
    Body = "{
    \"/list\": \"return a list of record IDs\",
    \"/get/ID\": \"retrieve a record by its ID\",
    \"/create\": \"create a new record; return its ID\",
    \"/update/ID\": \"update an existing record\",
    \"records_file_name\": \"~s\",
    \"state_file_name\": \"~s\",
}",
    Body1 = io_lib:format(Body, [Recordfilename, Statefilename]),
    {Body1, Req, State}.

get_help_text(Req, State) ->
    {ok, Recordfilename} = application:get_env(rest_update, records_file_name),
    {ok, Statefilename} = application:get_env(rest_update, state_file_name),
    Body = "
- list: return a list of record IDs~n
- get:  retrieve a record by its ID~n
- create: create a new record; return its ID~n
- update:  update an existing record~n
- records_file_name: ~s~n
- state_file_name: ~s~n
",
    Body1 = io_lib:format(Body, [Recordfilename, Statefilename]),
    {Body1, Req, State}.

generate_id() ->
    {ok, Statefilename} = application:get_env(rest_update, state_file_name),
    dets:open_file(state_db, [{file, Statefilename}, {type, set}]),
    Records = dets:lookup(state_db, current_id),
    Response = case Records of
        [{current_id, CurrentId}] ->
            NextId = CurrentId + 1,
            %    CurrentId, NextId]),
            dets:insert(state_db, {current_id, NextId}),
            Id = lists:flatten(io_lib:format("id_~4..0B", [CurrentId])),
            Id;
        [] ->
            error
    end,
    dets:close(state_db),
    Response.

7   Problems, enhancements, quibbles

  1. The handler opens and closes the DETS database for each operation. It might be nice to have a separate process which held the database open and responded to message to add a record, get a record, update a record, and get a list of all records. This process should be an OTP supervised process so that, if and when it dies, it will automatically be restarted.
  2. Better yet would be to start an Erlang process that "owns" the database resource. Doing so could give us a number of benefits: (1) The process could keep the DETS database open and would not need to open and close it for each request, as in our code above. (2) Because only one process which performs all database tasks would be running at any one time, the database access would in effect be serialized in a "critical section" of code, whereas in the above code it's likely possible to have a race condition that allows two request handler processes to interfere with each other's efforts to update the database. (3) If the database process were implemented as an OTP supervised process, it could be automatically restarted in the event of failure.

Published

Category

erlang

Tags

Contact