Sihl

Sihl is a batteries-included web framework built on top of Opium, Caqti, Logs and many more. Thanks to the modular architecture, included batteries can be swapped out easily. Statically typed functional programming with OCaml makes web development fun, fast and safe.

Getting Started

If you want to jump into code, have a look at the demo project. Otherwise keep reading and get started by creating your own app from scratch.

Prerequisites

You need to have OPAM installed and you should somewhat know OCaml and its tools. This guide walks you through setting up some development environment.

It is also recommended that you know OCaml's standard library and Lwt. This documentation explains a lot of concepts in terms of types, so you should be comfortable reading type signatures. If you are a beginner, check out the section OCaml for web development in Sihl to learn enough to be dangerous.

Installation

Sihl is distributed through OPAM, so go ahead and install OPAM. The easiest way to get started is to use the template that is always up-to-date.

App Generation

Run

git clone git@github.com:oxidizing/sihl.git

and

cp -r sihl/template app

where app is the directory of your app.

Directory Structure

Introduction

The default Sihl application structure is intended to provide a great starting point for both large and small applications. However, you are free to organize your application in whichever way you like.

├── app
│   ├── command
│   ├── domain
│   ├── schedule
├── database
├── logs
├── public
├── resources
├── routes
│   └── routes.ml
├── run
│   └── run.ml
├── service
│   └── service.ml
├── test
└── web
    ├── handler
    ├── middleware
    └── view

The Root directory

The App Directory

The app directory contains the core code of your application. We will explore this directory in more detail soon. However, almost all of your application code will be in this directory.

The Database Directory

The database directory contains your database migrations and seeds.

The Logs Directory

The logs directory contains your application logs, by default app.log and error.log.

The Public Directory

The public directory is served by the HTTP server of Sihl. It is the target directory and it usually contains built CSS, JavaScript and assets.

The Resources Directory

The resource directory contains the source code of the static assets that are served from the public directory. Put the source code of your any JavaScript projects here for instance.

The Routes Directory

The routes directory contains all of the route definitions for your application. By default, two routers are created: One for sites and one for JSON APIs.

The site router is using sessions, CSRF protection and flash messages. If your application doesn't have a JSON API then it is likely that all of your routes will be here.

The api router contains routes that are intended to be stateless, and are using tokens and request limiting.

The Run Directory

The run directory is the main entry point of the executable that is your Sihl app. The run.ml file knows all other modules and it is the single place where you wire up your app.

In the file run.ml, you register the services listed in service.ml. Sihl doesn't know about the services in service.ml, so you must register the services that you want Sihl to start here.

A run.ml setup when using PostgreSQL can look like this:

let commands = [ Command.Add_todo.run ]

let services =
  [ Service.Migration.register ~migrations:Database.Migration.all ()
  ; Service.Token.register ()
  ; Service.EmailTemplate.register ()
  ; Service.MarketingEmail.register ()
  ; Service.TransactionalEmail.register ()
  ; Service.User.register ()
  ; Service.PasswordResetService.register ()
  ; Sihl.Schedule.register ()
  ; Sihl.Web.Http.register ~middlewares Routes.router
  ]
;;

let () = Sihl.App.(empty |> with_services services |> run ~commands)

Run the executable to run the Sihl app, which will start the registered services.

The Service Directory

The service directory contains Sihl services that you can use in your application.

Most of of Sihl's features are provided as services.

Services can have dependencies on each other. For instance, the Sihl.Database.Migration service depends on the Sihl.Database service, since a database connection is requied to run migrations.

The service.ml file contains a list of modules of services that you can use in your project. This is where you decide the service implementation.

module Migration = Sihl.Database.Migration.PostgreSql

This is also where you have to list services of Sihl packages which are not contained in the sihl package like sihl-user and sihl-token.

module Migration = Sihl.Database.Migration.PostgreSql
module User = Sihl_user.PostgreSql
module Token = Sihl_token.JwtPostgreSql
module EmailTemplate = Sihl_email.Template.PostgreSql
module MarketingEmail = Sihl_email.SendGrid
module TransactionalEmail = Sihl_email.Smtp

In service.ml, you also build your own services with module functors. This concept gives Sihl its modularity, which allows you to easily create your own services.

Services can be passed an optional service context ctx. The service context is of type (string * string) list. By default, the context is empty. It allows you to pass read-only settings to services that are valid for just one service call. It is up to the service implemenation to use the context, different service implementations might read different parts of the context.

module Token = Sihl_token.JwtPostgreSql
module PasswordResetService = Sihl_user.Password_reset.MakePostgreSql (Token)

In the example above Sihl_user.Password_reset.MakePostgreSql is a functor that takes a token service to instantiate a password reset service.

The Test Directory

The test directory contains unit and service tests.

The Web Directory

The web directory contains middlewares, HTML and JSON views and handlers. We have a closer look at these concepts at the basics.

The Handler Directory

The handler directory contains your handlers and controllers. Here you take care of parsing the requests, calling application logic and creating responses.

The Middleware Directory

The middleware directory contains your own Rock middleware implementations. Many of Sihl's features like CSRF tokens, session handling and flash messages are implemented as middlewares.

The View Directory

The view directory contains HTML and JSON views that render response bodies.

The App Directory

The Command Directory

The command directory contains custom CLI commands that can be executed alongside the built-in commands. Next to HTTP, this is the other way to interact with your app.

The Domain Directory

The domain contains your application logic as models. You are free to structure your code however you like. It is not possible to depend on any of the web modules in here.

We suggest that an approach that is inspired by Domain-Driven-Design. You start with a service, a entity.ml and a repository.ml for every model. The service is named after the model. Let's say the model is called app/domain/shopping, then your service is app/domain/shopping/shopping.ml. Once you identify other models, you can extract them. As an example you could extract a model app/domain/customer. Grow your app in terms of models and be mindful about the dependencies between them.

The entity.ml file contains types and pure business logic. You are not allowed to do I/O like network requests here. Try to have as much as of your application as possible in entities, as they are easy to test and understand.

The repository.ml file contains database queries and helpers. You can have your database types which might differ from the business types defined in entity.ml.

The service file contains code that glues pure business logic in entity.ml to the impure repository.ml. The service exposes a public API that other domains and services can use. It is not allowed to use repositories of other services directly.

Sihl provides you with the surrounding infrastructure services like session handling, user management, job queues and many more.

The app in app/domain should not depend on infrastructure services if possible. A well designed app will run with other web frameworks after minimal adjustments.

The Schedule Directory

The schedule directory contains schedules (or crons jobs) that run periodically.

Configuration

One of the design goals of Sihl is safety. A Sihl app does not start if the required configurations are not present. You can get a list of required configurations with the command make sihl config:list. Note that the list of configurations depends on the services that are installed.

Providing Configuration

There are three ways to provide configurations:

• environment variables
• .env files
• programmatically using Sihl.Configuration.store

.env files have to be placed in the project root directory. Sihl tries to find out where that is. In general, the root directory is the directory that is under version control, i.e. where the .git directory is. You can override the project root with ROOT_PATH. You can set the location of your .env file with ENV_FILES_PATH if you want to move it away from the project root.

Reading Configuration

Use Sihl.Configuration to read configuration. You can also use it to programmatically store some configuration.

Examples:

let smtp_host = Sihl.Configuration.read_string "SMTP_HOST" in

The Basics

Everything regarding web lives in Sihl.Web.

Routing

The routes are the HTTP entry points to your app. They describe what can be done in a declarative way.

Routes can be created with Sihl.Web:

let list_todos = Sihl.Web.get "/" Handler.list
let add_todos = Sihl.Web.post "/add" Handler.add
let order_pizza = Sihl.Web.post "/order" Handler.order

A route takes a path and a handler, the HTTP method is given by the function.

The routes live in the root directory routes/routes.ml. A list of routes can be mounted under a path (called scope) with a middleware stack. The site routes for instance are mounted like:

let router =
  Sihl.Web.combine
    ~middlewares
    ~scope:"todos"
    [ list_todos; add_todos; order_pizza ]
;;

This creates a Sihl.Contract.Http.router. Routers are passed to the web server when registering the service Sihl.Web.Http. When you run the Sihl app, Sihl starts the HTTP server serving the registered route.

let router =
  Sihl.Web.choose
    [Routes.api_router; Routes.site_router]
;;

let services = [ Sihl.Web.Http.register router ]

Routers can be composed arbitrarily.

let router =
  Sihl.Web.(
    choose
      ~middlewares:[ main ]
      [ choose
          ~middlewares:[ admin ]
          ~path:"admin"
          [ get "users" list_users
          ; post "users" add_users
          ; post "users/:id" update_user
          ]
      ; choose
          ~path:"orders"
          [ get "" list_orders
          ; post "" add_order
          ; post ":id" update_order
          ; get ":id" show_order
          ]
      ])
;;

Note that the middlewares are only triggered for a request that matches a route. GET /admin/users passes the main and admin middleware, while GET /admin/foo does not. Middlewares that should be triggered for every request (including those that don't have a route) have to be passed directly to Sihl.Web.Http.register. these middlewares are also called global.

Requests

In backend web development, everything starts with an HTTP request. Sihl uses Opium (which uses Rock) under the hood. Your job is to create a Rock.Response.t given a Rock.Request.t.

This is done using a handler, which has the signature val handler : Rock.Request.t -> Rock.Response.t Lwt.t. In the handler you call your own code or Sihl services. A handler looks like this:

let list req =

  let csrf = Sihl.Web.Csrf.find req in
  let notice = Sihl.Web.Flash.find_notice req in
  let alert = Sihl.Web.Flash.find_alert req in
  let%lwt todos, _ = Todo.search 100 in
  Lwt.return @@ Opium.Response.of_html (Template.page csrf todos alert notice)
;;

A request has the following lifecycle:

• HTTP request: The HTTP server receives a request
• Gloabal middlewares in: The request goes through a list of global middlewares
• Route: The request either matches one of the routes or it doesn't
• Scoped middlewares in: If there is a match, the request goes through a list of middlewares
• Handler: If there was a match, the request reaches a handler and triggers service calls which yields a response
• Scoped middlewares out: If there was a match, the response goes through a list of scoped middlewares
• Global middlewares out: The response goes through a list of global middlewares
• HTTP response: The response is sent back to the client

In order to learn more about the request lifecycle, check out Opium examples.

To deal with requests, you can use Sihl.Web.Request which is just an alias for Opium.Request.

Responses

To deal with responses, you can use Sihl.Web.Response which is just an alias for Opium.Response.

Views

Sihl makes no assumptions about how you create HTML and JSON responses, so you are free to use whatever you like. However, if you want to work with generators it can be helpful to understand the tools and conventions they use.

HTML

Use TyXML to generate HTML in a type-safe way.

JSON

Use ppx_yojson_conv to derive JSON encoders for your types.

Middleware

You have seen that a handler is a function with the signature val handler : Rock.Request.t -> Rock.Response.t Lwt.t. A middleware is a function with the signatures val middleware : handler -> handler.

Middlewares are used to wrap handlers and add functionality to them.

Sihl middlewares live in Sihl.Web.Middleware, go ahead and have a look to get an idea what middlewares can do.

Sihl.Web is built on top of Opium which allows you to use all middlewares shipped with Opium.

Custom middlewares

Have a look at this example on how to build an Opium middleware. You can put your own Opium middlewares in web/middleware.

Default stacks

A middleware stack is a chain of middlewares. A request has to go through the chain before your handler is called. By default, Sihl creates two routers with default middleware stacks.

In route/site.ml:

let middlewares =
  [ Sihl.Web.Middleware.id ()
  ; Sihl.Web.Middleware.error ()
  ; Opium.Middleware.content_length
  ; Opium.Middleware.etag
  ; Sihl.Web.Middleware.static_file ()
  ; Sihl.Web.Middleware.flash ()
  ]
;;

let handlers = (* Your handlers that return HTML responses *)

let router = Sihl.Web.Http.router ~middlewares ~scope:"/api" handlers

In route/api.ml

let middlewares =
  [ Sihl.Web.Middleware.id ()
  ; Sihl.Web.Middleware.error ()
  ]
;;

let handlers = (* Your handlers that return JSON responses *)

let router = Sihl.Web.Http.router ~middlewares ~scope:"/site" handlers

CSRF Protection

Introduction

Cross-Site Request Forgery (CSRF) is an attack that forces an end user to execute unwanted actions on a web application in which they’re currently authenticated. With a little help of social engineering (such as sending a link via email or chat), an attacker may trick the users of a web application into executing actions of the attacker’s choosing. If the victim is a normal user, a successful CSRF attack can force the user to perform state changing requests like transferring funds, changing their email address, and so forth. If the victim is an administrative account, CSRF can compromise the entire web application. (https://owasp.org/www-community/attacks/csrf)

Installation

Add Sihl.Web.Middleware.csrf to your list of middlewares. This middleware is enabled by default for site routes in routes/site.ml.

let middlewares = [ Sihl.Web.Middleware.csrf () ]

Adding Sihl.Web.Middleware.csrf to global middlewares is not recommended. Global middlewares are injected and run before any other middlewares, which is sometimes not desirable for CSRF protection. Additionally, there might be some routes where CSRF protection is unnecessary and sometimes a hinderance (for non-state changing POST requests for example).

Usage

Every form that uses POST needs to have a CSRF token associated to it. Use a hidden field <input type="hidden" name="csrf" value=.../> in all your forms.

In a handler you can fetch the CSRF token with Sihl.Web.Csrf.find and pass it to the view.

let form req =
  let csrf = Sihl.Web.Csrf.find req in
  Lwt.return @@ Sihl.Web.Response.of_html (View.some_form csrf)

During development and testing the CSRF check is disabled to make it easier to debug requests. In order to enable CSRF checks you can set CHECK_CSRF to true.

Session

Introduction

Since HTTP driven applications are stateless, sessions provide a way to store information about the user across multiple requests. That user information is typically placed in a persistent store/backend which can be accessed from subsequent requests. (https://laravel.com/docs/8.x/session#introduction)

Sihl ships with a cookie-based session implementation.

Usage

Use Sihl.Web.Session to handle sessions.

In order to set session values create a response first:

let handler _ =
  let resp = Opium.Response.of_plain_text "some response" in
  Lwt.return @@ Sihl.Web.Session.set [("user_id", Some "4882")] resp

Note that setting the session like this overrides any previously set sessions. You should not chain Sihl.Web.Session.set or touch the session cookie manually.

To read a session value:

let handler req =
  let user_id = Sihl.Web.Session.find "user_id" req in
  match user_id with
  | Some user_id -> (* fetch user and do stuff *)
  | None -> Opium.Response.of_plain_text "User not logged in"

Limitations

The cookie-backed session stores the session content in a cookie that is signed and sent to the client. The session content can be inspected by the client and the maximum data size is 4kb.

If you need to store sensitive data or values larger than 4kb, use sihl-cache as a generic persistent key-value store and just store a reference in the actual session. A common use case is to store the user id in the cookie and the associated values in the cache.

Error Handling

Introduction

Exceptions happen and they should be dealt with. They are not recoverable and should be logged and raised. The user gets to see a nice 500 error page.

Installation

Add Sihl.Web.Middleware.error to your list of middlewares. Check out the documentation to learn how to install custom error reporters. The error middleware is enabled by default for site and JSON routes in routes/site.ml and routes/api.ml.

let middlewares = [ Sihl.Web.Middleware.error () ]

Logging

Sihl uses Logs for log reporting and log formatting. It is highly recommended to have a look at the basics if you intend to customize logging.

Set the log level using LOG_LEVEL to either error, warning, debug or info.

Reporters

By default Sihl has two log reporters: CLI and file.

The CLI reporter logs colored output to stdout and stderr and the file reporter logs to the logs directory.

You can install custom log reporters if you want to stream logs to some logging service for instance. In run/run.ml:

let my_log_reporter = (* streaming log reporter *)
let () = Sihl.App.(empty |> with_services services |> run ~log_reporter:my_log_reporter)

Logging

Use Logs to actually log things. We recommand to create a custom log source for every service.

let log_src = Logs.Src.create "booking.orders"

module Logs = (val Logs.src_log log_src : Logs.LOG)

Logs.err (fun m -> m "This prevents the program from running correctly");
Logs.warn (fun m -> m "This is a suspicious condition that might lead to the program not running correctly");
Logs.info (fun m -> m "This allows the program user to understand what is going on. If your program is a pure HTTP app, you probably don't need this log level.")
Logs.debug (fun m -> m "This is for programmers to understand what is going on.")

REST

Sihl provides built-in support to quickly build RESTful web apps. This feature is inspired by Rails.

Introduction

With Sihl.Web.Rest you can make apps and services accessible through the web. A resource pizzas can have following HTTP routes. Each route is associated to an action.

GET       /pizzas          `Index  Display a list of all pizzas
GET       /pizzas/new      `New    Return an HTML form for creating a new pizza
POST      /pizzas          `Create Create a new pizza
GET       /pizzas/:id      `Show   Display a specific pizza
GET       /pizzas/:id/edit `Edit   Return an HTML form for editing a pizza
PATCH/PUT /pizzas/:id      `Update Update a specific pizza
DELETE    /pizzas/:id      `Delete Delete a specific pizza

Model

A resource model is represented as a type and combinators that have some business logic. This is not specific to Sihl.

type pizza =
  { name : string
  ; is_vegan : bool
  ; price : int
  ; created_at : Ptime.t
  ; updated_at : Ptime.t
  }

Schema

A schema connects the static world of OCaml types and the dynamic world of forms and urlencoded values. Sihl uses Conformist to decode and validate data.

Attach your own validators to schema fields to validate form input elements with business logic.

let create_ingredient name is_vegan price = ...

let[@warning "-45"] pizza_schema
    : (unit, string -> bool -> int -> pizza, pizza) Conformist.t
  =
  Conformist.(
    make
      Field.
        [ string
            ~validator:(fun name ->
              if String.length name > 12
              then Some "The name is too long, it has to be less than 12"
              else if String.equal "" name
              then Some "The name can not be empty"
              else None)
            "name"
        ; bool "is_vegan"
        ; int
            ~validator:(fun price ->
              if price >= 0 && price <= 10000
              then None
              else Some "Price has to be positive and less than 10'000")
            "price"
        ]
      create_pizza)
;;

Opening the module Conformist shadows the operator :: which is used to create lists by consing elements to an empty list. Conformist overwrites this operator which allows you to use the list syntax [el1; el2; ...] to create schemas. Depending on your setup, dune warns you that the thing you are creating is not really a list.

This is on purpose and you can suppress the warning with [@warning "-45"].

CRUD Service

A CRUD service creates, reads, updates and deletes models. Implement the Sihl.Web.Rest.SERVICE interface.

View

The last component is the view. A view is a module of type Sihl.Web.Rest.VIEW. It receives CSRF tokens, the resource model, form data and a request and returns HTML.

Routers

The two basic building blocks to build resources are a model and a schema.

Service

Create a resource by implementing Sihl.Web.Rest.SERVICE and Sihl.Web.Rest.VIEW.

The model, schema, service and view are combined using Sihl.Web.Rest.resource.

let pizzas =
  Sihl.Web.(
    choose
      ~middlewares:[ Middleware.csrf (); Middleware.flash () ]
      (Rest.resource_of_service
         ~only:[ `Index; `Show; `New; `Destroy ]
         "pizzas"
         Pizza.schema
         (module Pizza : Rest.SERVICE with type t = Pizza.t)
         (module View.Pizza : Rest.VIEW with type t = Pizza.t)))
;;

Don't forget to apply the CSRF and flash middlewares for the REST routes, this is a requirement. The pizzas router can be passed to the Sihl.Web.Http service before starting the Sihl app.

You have to assign the types t of your service and view to your resource model type.

To specify which actions to support per resource, use the only argument. Routes of actions that are not listed return 404.

Controller

If you need more control over a resource, implement your own Sihl.Web.Rest.CONTROLLER. This approach is quite low-level and you have to take care of all the wiring like setting flash messages, building resource paths and redirecting.

let pizzas =
  Sihl.Web.(
    choose
      ~middlewares:[ Middleware.csrf (); Middleware.flash () ]
      (Rest.resource_of_controller
         ~only:[ `Index; `Show; `New; `Destroy ]
         "pizzas"
         Pizza.schema
         (module Pizza : Rest.CONTROLLER with type t = Pizza.t)))
;;

Database

Most Sihl services have MariaDB and PostgreSQL backends, SQLite is planned as well. However, Sihl doesn't make any assumptions about the persistence layer so you are free to bring your own tools and libraries. Sihl uses Caqti under the hood, which provides a common abstraction on top of SQL databases.

The Sihl.Database module provides functions for querying the database, running database schema migrations and it deals with connection pooling.

Query Interface

The database service creates and manages a connection pool. Configure the pool size with DATABASE_POOL_SIZE, the default is 10. Connection pools are used to decrease latency by keeping datbase connections open.

Provide a DATABASE_SKIP_DEFAULT_POOL_CREATION if you want to manage database pools yourself. This is useful if you need multiple databases. The default assumption of Sihl is to have one application database.

Use DATABASE_CHOOSE_POOL to use a specific named pool that was created using Sihl.Database.add_pool to run commands.

The main functions to run queries on the connection pool are Sihl.Database.find, Sihl.Database.find_opt, Sihl.Database.exec, Sihl.Database.collect. Use these to run caqti requests directly on the connection pool.

let find_request =
  let open Caqti_request.Infix in
  {sql|
    SELECT
      cache_value
    FROM cache
    WHERE cache.cache_key = ?
  |sql} |> Caqti_type.(string ->? string)
;;

let find key = Sihl.Database.find_opt find_request key
;;

If you need to run multiple caqti requests on the same connection, use Sihl.Database.query.

let request1 = (* ... *)
let request2 = (* ... *)
;;

let find key =
  Sihl.Database.query (fun (module Connection : Caqti_lwt.CONNECTION) ->
      let%lwt result1 = Connection.find_opt request1 key |> Lwt.map Sihl.Database.raise_error in
      let%lwt result2 = Connection.find_opt request2 key |> Lwt.map Sihl.Database.raise_error
      Lwt.return (result1, result2))

Context

The database implementations accept a pool identifier in the pool key of the context. You can choose which database pool should be used for each call.

Make sure to initialize the pools using Sihl.Database.add_pool before you reference them.

Transactions

Use Sihl.Database.transaction and Sihl.Database.transaction' to run queries in a database transaction.

let query_with_transaction _ () =
  let%lwt usernames =
    Sihl.Database.query (fun connection ->
        let%lwt () = drop_table_if_exists connection in
        let%lwt () = create_table_if_not_exists connection in
        Sihl.Database.transaction (fun connection ->
            let%lwt () = insert_username connection "foobar trx" in
            get_usernames connection))
  in
  let username = List.find (String.equal "foobar trx") usernames in
  Alcotest.(check string "has username" "foobar trx" username);
  Lwt.return ()
;;

Migrations

Migrations live in database/migration.ml. Use make sihl migrate to run pending migrations to update the database schema. The API is can be found at Sihl.Contract.Migration.Sig.

Services that you register can install their own migrations. It is important to run make sihl migrate after installing a new service with a SQL database backend.

Seeding

Seeds are used to set the state of a database to allow development with test data or to run automated tests.

Seeds live in database/seed.ml. Unlike in other web frameworks, Sihl seeds are just function calls. This means that you can not export the current database state as seeds and you have to manually write them. Your seeds are using public service API which doesn't break often. This allows for seeding an app that uses many different service backends. Also, seeding doesn't depend on the data model.

Often you want to run seeds before doing a development step. Use commands to run seeds from the CLI.

Testing

OCaml catches many bugs at compile-time and Sihl enforces certain invariants at start-up time. Howver, there are still many bugs out there that need to be caught. Automated tests can be a great tool to complement the safety of OCaml and Sihl.

Have a look at this introduction to test-driven development in OCaml.

Sihl uses Alcotest as a test runner.

Arrange-Act-Assert

Structure your tests using Arrange-Act-Assert.

Arranging your state requires you to clean up first. You don't have direct access to remove the state of infrastructure services provided by Sihl or Sihl packages such as sihl-user. In order to clean their state you can use Sihl.Cleaner.clean_all.

let create_list_and_do _ () =
  let open Todo.Model in
  let%lwt () = Sihl.Cleaner.clean_all () in
  let%lwt _ = Todo.create "do laundry" in
  let%lwt _ = Todo.create "hoover" in
  let%lwt todos = Todo.search 10 in
  let t1, t2 =
    match todos with
    | [ t1; t2 ], n ->
      Alcotest.(check int "has 2" 2 n);
      t1, t2
    | _ -> Alcotest.fail "Unexpected number of todos received"
  in
  Alcotest.(check string "has description" "hoover" t1.description);
  Alcotest.(check string "has description" "do laundry" t2.description);
  let%lwt () = Todo.do_ t1 in
  let%lwt t1 = Todo.find t1.id in
  Alcotest.(check bool "is done" true (Todo.is_done t1));
  Lwt.return ()
;;

If you have complex pre-conditions, you should move the service calls to the database directory and create seeds out of them. Parametrize the seeds as needed for re-use in other tests.

Digging Deeper

Pagination & Search

Displaying things and collections of things is a common thing in web development. Often we don't want to display all the items that are stored, but present the user a nice interface to search collections.

Sihl services usually follow a convention where they expose a function

val search:
  ?sort:[ `Asc | `Desc ] ->
  ?filter:string ->
  ?limit:int ->
  ?offset:int ->
  unit ->
  (t list * int) Lwt.t

that can be used to fetch, sort and filter a partial view on the whole collection of t.

To implement your own function with this signature, use Sihl.Database.prepare_search_request and Sihl.Database.run_search_request:

let filter_fragment =
  {sql|
      WHERE user_users.email LIKE $1
        OR user_users.username LIKE $1
        OR user_users.status LIKE $1 |sql}
;;

(* We need to escape this SQL query because
   it breaks syntax highlghting of the documentation *)
let search_query =
  {sql|
      SELECT
        COUNT(\*\) OVER() as total,
        uuid,
        email,
        username,
        password,
        status,
        admin,
        confirmed,
        created_at,
        updated_at
      FROM user_users |sql}
;;

let request =
  Sihl.Database.prepare_search_request
    ~search_query
    ~filter_fragment
    ~sort_by_field:"id"
    user
;;

let search sort filter ~limit ~offset =
    Sihl.Database.run_search_request
      connection
      request
      sort
      filter
      ~limit
      ~offset
;;

The first column of the search_query needs to be the total number of rows after applying the filter. The total number is higher than the limit, if there are a lot of rows.

Note that if you need features like sorting on multiple columns or complex filters, you have to take care of safely building your SQL queries at runtime. The approach shown above is safe from SQL injection, fast (due to prepared statements) and simple to use.

Using offset-based pagination comes with some drawbacks that you should be aware of.

Compiling assets

The project template has a asset pipeline set up that watches files in the resource directory and compiles assets into the public directory. Run make assets to compile the assets once and make assets_watch for watch for file changes.

ParcelJS is used to have a zero configuration experience when dealing with all sorts of assets types.

By default, Sihl serves static files from the public directory.

File Storage

Retrieving files

Sihl serves the public directory under the path /assets by default using Sihl.Web.Middleware.static_file.

You can configure the directory to be served using PUBLIC_DIR. The URI prefix can be configured using PUBLIC_URI_PREFIX.

Uploading files

This feature is not implemented in Sihl yet. Use this Opium example meanwhile.

Commands

Introduction

There are two ways to interact with a Sihl app, via HTTP and via the command line interface (CLI) commands. Sihl has built-in support for both. In fact, it is often better to implement CLI commands before creating the routes, handlers and views. Commands are a great way to quickly call parts of the app with parameters.

Commands are handled with the module Sihl.Command.

Built-in commands

Run make sihl if you used Spin to create the app, otherwise execute the Sihl app exectuable.

2021-04-10T13:18:34-00:00 [INFO] [sihl.core.app]: Setting up...
2021-04-10T13:18:34-00:00 [INFO] [sihl.core.configuration]: SIHL_ENV: development

Sihl

Run one of the following commands with the argument "help" for more information.

config
gen.html
gen.json
gen.model
gen.view
migrate
random
routes
server
user.admin

This is the list of built-in commands. Whenever you install a package and register Sihl services you get access to more commands.

Custom commands

Create a file in the directory app/command to create custom commands.

let run =

  Sihl.Command.make
    ~name:"todo.add"
    ~usage:"<todo description>"
    ~description:"Adds a new todo to the backlog"
    (fun args ->
      match args with
      | [ description ] ->
        let%lwt _ = Todo.create description in
        Lwt.return (Some ())
      | _ -> Lwt.return None)
;;

Don't forget to pass the list of commands in run/run.ml when starting the app:

let () =
  Sihl.App.(
    empty |> with_services services |> run ~commands:[ Command.Add_todo.run ])
;;

Run make sihl to see your custom command added to the list of registered commands.

2021-04-10T13:18:34-00:00 [INFO] [sihl.core.app]: Setting up...
2021-04-10T13:18:34-00:00 [INFO] [sihl.core.configuration]: SIHL_ENV: development

Sihl

Run one of the following commands with the argument "help" for more information.

config
gen.html
gen.json
gen.model
gen.view
migrate
random
routes
server
todo.add
user.admin

Generators

Sihl has built-in generators gen.model, gen.view and gen.html. Similar to Rails they help you generating resources, services, repositories and HTML views. Unlike Rails, Sihl doesn't have an abstraction layer for data access (such as an ORM). You should be familiar with SQL.

The generators create files and folders before printing instructions for manual steps to finalize the generation. gen.html can create a fully functional CRUD web interface from a schema.

Generators are CLI commands just like migrate, run the generator commands without any arguments to display help text describing its usage.

Service

$ make sihl gen.model mariadb order order_number:int premium:bool price:int location:string delivery_date:datetime
2021-04-12T16:20:46-00:00 [INFO] [sihl.core.app]: Setting up...
2021-04-12T16:20:46-00:00 [INFO] [sihl.core.configuration]: SIHL_ENV: development
Wrote file 'sihl-demo/app/domain/order/order.ml'
Wrote file 'sihl-demo/app/domain/order/order.mli'
Wrote file 'sihl-demo/app/domain/order/entity.ml'
Wrote file 'sihl-demo/app/domain/order/repo.ml'
Wrote file 'sihl-demo/app/domain/order/dune'
Wrote file 'sihl-demo/test/order/test.ml'
Wrote file 'sihl-demo/test/order/dune'
Wrote file 'sihl-demo/database/order.ml'
Command 'gen.model' ran successfully in 1.614ms

gen.model generates a model consisting of a service, an entity, a reposiory, migrations, and some simple CRUD tests. Note that you have to specify the model name and the database you are using. You can run the tests right after generation.

The created model type contains an id of UUID V4 as string, created_at as Ptime.t and updated_at as Ptime.t.

View

$ make sihl gen.view order order_number:int premium:bool price:int location:string delivery_date:datetime
2021-04-12T16:25:48-00:00 [INFO] [sihl.core.app]: Setting up...
2021-04-12T16:25:48-00:00 [INFO] [sihl.core.configuration]: SIHL_ENV: development
Wrote file 'sihl-demo/web/view/order/dune'
Wrote file 'sihl-demo/web/view/order/view_order.ml'
Command 'gen.view' ran successfully in 1.527ms

gen.view generates an HTML view for a resource. Note that you don't have to specify a database.

HTML Resource

$ make sihl gen.html mariadb order order_number:int premium:bool price:int location:string delivery_date:datetime
2021-04-13T07:19:08-00:00 [INFO] [sihl.core.app]: Setting up...
2021-04-13T07:19:08-00:00 [INFO] [sihl.core.configuration]: SIHL_ENV: development
Wrote file 'sihl-demo/app/domain/order/order.ml'
Wrote file 'sihl-demo/app/domain/order/order.mli'
Wrote file 'sihl-demo/app/domain/order/entity.ml'
Wrote file 'sihl-demo/app/domain/order/repo.ml'
Wrote file 'sihl-demo/app/domain/order/dune'
Wrote file 'sihl-demo/test/order/test.ml'
Wrote file 'sihl-demo/test/order/dune'
Wrote file 'sihl-demo/database/order.ml'
Wrote file 'sihl-demo/web/view/order/dune'
Wrote file 'sihl-demo/web/view/order/view_order.ml'

Resource 'orders' created.

Copy this route

    let order =
      Sihl.Web.choose
        ~middlewares:
          [ Sihl.Web.Middleware.csrf ()
          ; Sihl.Web.Middleware.flash ()
          ]
        (Rest.resource
          "orders"
          Order.schema
          (module Order : Rest.SERVICE with type t = Order.t)
          (module View_order : Rest.VIEW with type t = Order.t))
    ;;

into your `routes/routes.ml` and mount it with the HTTP service. Don't forget to add 'order' and 'view_order' to routes/dune.

Add the migration

    Database.Order.all

to the list of migrations before running `sihl migrate`.
You should also run `make format` to apply your styling rules.

Command 'gen.html' ran successfully in 1.964ms

gen.html generates a service, a model, a reposiory, simple CRUD tests and an HTML view. Note that you have to specify the service name and the database you are using. You need to do the steps listed manually in order to test the resource. Once your project compiles, run the tests and browse the root path of the resources to use the fully functional CRUD form.

JSON Resource

This is not yet implemented. Please open an issue if you need this feature.

Randomness

Documentation is in the making, check out Sihl.Random meanwhile.

Time

Documentation is in the making, check out Sihl.Time meanwhile.

Scheduling

Documentation is in the making, check out Sihl.Schedule meanwhile.

OCaml for Sihl

This section will not tell you all about OCaml but instead give some pointers on where to look things up and list some conventions in Sihl.

Basics

After studying the basics, you should learn about data and higher-order programming in order to manipulate data.

Once you feel comfortable with these concepts, go ahead and read about the remaining Sihl-specific topics.

Error handling

A good primer can be found here.

On top of the general error handling patterns in OCaml, there is a convention in Sihl services that you are going to use. Some services return (unit, string) Result.t Lwt.t while others return unit Lwt.t and raise an exception.

Exception and Option

Lets look at a function that returns a user given an email address.

(** [find_by_email email] returns a [User.t] if there is a user with an [email]
    address. Raises an [{!Exception}] if no user is found. *)
val find_by_email : string -> t Lwt.t

This function raises an exception if no user was found. But this function can also raise an exception if the connection to the database is broken. These two cases are different, but both raise exceptions.

If you get the email address from the end user directly, you might want to use this function instead.

(** [find_by_email_opt email] returns a [User.t] if there is a user with email
    address [email]. *)
val find_by_email_opt : string -> t option Lwt.t

If there was no user found, you get None back and you can ask your user for the correct email address. This function still raises if the database connection breaks. The failing database connection is not the user's fault, it can not be recovered by the user doing something else. This is an issue with our infrastructure or our code. The best thing to do here is to let the service raise an exception and let the error middleware handle it with a nice 500 error page.

Use exceptions for errors that are not the fault of a user. The variant find_by_email is included for convenient internal usage, when you want to send an email to a list of users in a bulk job for instance.

Result

Let's take a look at following function:

(** [update_password user ~old new] sets the [new] password of the [user] if the current password matches [old]. *)
val update_password : User.t -> ~old:string -> string -> (unit, string) Result.t Lwt.t

In this case, the function returns an error with an error message if the provided password is wrong. Why can't we just return unit option Lwt.t and just act on None if something is wrong?

We want to distinguish various invalid user inputs. The user might provide an old password that doesn't match the current one, but the user might also provide a password that is not long enough according to some password policy. In both cases, the user needs to fix the error so we show them the message.

Lwt

Sihl is built on top of the Lwt library, which is similar to Promises in other languages. From the web module to the migration service, everything uses Lwt so it is crucial to understand the basic API and usage.

Sihl uses lwt_ppx which makes it easy to deal with Lwt.t and it gives you better error messages.

let add req =
  match Sihl.Web.Form.find_all req with
  | [ ("description", [ description ]) ] ->
    let%lwt _ = Todo.create description in
    let resp = Opium.Response.redirect_to "/" in
    let resp = Sihl.Web.Flash.set_notice (Some "Successfully updated") resp in
    Lwt.return resp
  | _ ->
    let resp = Opium.Response.redirect_to "/" in
    let resp = Sihl.Web.Flash.set_alert (Some "Failed to update todo description") resp in
    Lwt.return resp
;;

Todo.create creates a todo with a description and it returns unit Lwt.t on success. In order to keep the code simple, use let%lwt. If you use let%lwt, you have to return 'a Lwt.t, so the last expression has to have an Lwt.

Build system

Sihl uses dune as a build system. If you are using the Spin template, the most common commands are listed in the Makefile. However, since you are in charge of your domain and its directory structure, you should become familiar with the basics of dune.

The Quickstart should cover most of it.

API

• Sihl.App
• Sihl.Cleaner
• Sihl.Command
• Sihl.Contract
• Sihl.Database
• Sihl.Configuration A module to manage service configurations.
• Sihl.Container A module to manage the service container and service lifecycles.
• Sihl.Log
• Sihl.Random
• Sihl.Schedule
• Sihl.Time
• Sihl.Web