Legend:
Page
Library
Module
Module type
Parameter
Class
Class type
Source
Source file file_manager_intf.ml
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313(*
* Copyright (c) 2022-2022 Tarides <contact@tarides.com>
*
* Permission to use, copy, modify, and distribute this software for any
* purpose with or without fee is hereby granted, provided that the above
* copyright notice and this permission notice appear in all copies.
*
* THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
* WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
* ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
* WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
* ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
* OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*)openImportmoduletypeS=sig(** Abstraction that governs the lifetime of the various files that are part
of a pack store (except the branch store).
The file manager (FM) handles the files one by one and makes explicit all
the interactions between them (except the index which is handled from a
high level API). This allows to gain confidence on SWMR consistency and
crash consistency.
{2 Two types of guarantees}
Irmin_pack_unix supports the SWMR access scheme. It means that it is
undefined for the files to be opened twice in RW mode by 2 FMs. It also
means that it is undefined for a FM in RW mode to be used simultaneously
from 2 different fibers.
Irmin_pack_unix aims to be (crash) consistent (in the ACID sense). In case
of a system failure (e.g. power outage), the files should be left in a
consistent state for later uses.
Ensuring SWMR consistency is easier than ensuring crash consistency
because the of the OS's shared page cache.
{2 Files mutations}
Here are all the moments where the files managed may be mutated:
- 1. During [create_rw].
- 2. During [open_rw] if a major version upgrade is necessary.
- 3. During the flush routines in file_manager.
- 4. During a GC, from the worker.
- 5. At the end of a GC, from the RW fiber, in [swap].
- 6. During integrity check routines.
1. 2. and 6. don't support SWMR and leave the store in an undefined state
in case of crash.
4. operates on files private to the worker. It doesn't necessitate to
worry about crash concistency and SWMR.
3. and 5. are highly critical. *)moduleIo:Io.SmoduleControl:Control_file.UpperwithmoduleIo=IomoduleDict:Append_only_file.SwithmoduleIo=IomoduleSuffix:Chunked_suffix.SwithmoduleIo=IomoduleIndex:Pack_index.SmoduleErrs:Io_errors.SwithmoduleIo=IomoduleSparse:Sparse_file.SwithmoduleIo=IomoduleLower:Lower.SwithmoduleIo=Iotypet(** A series of getters to the underlying files managed. *)valcontrol:t->Control.tvaldict:t->Dict.tvalsuffix:t->Suffix.tvalindex:t->Index.tvalprefix:t->Sparse.toptionvallower:t->Lower.toptiontypecreate_error:=[Io.create_error|Io.write_error|Io.open_error|Io.mkdir_error|`Corrupted_mapping_fileofstring|`Corrupted_control_fileofstring|`Double_close|`Unknown_major_pack_versionofstring|`Volume_missingofstring|`Not_a_directoryofstring|`Multiple_empty_volumes|`Index_failureofstring|`Sys_errorofstring|`No_tmp_path_provided]valcreate_rw:overwrite:bool->Irmin.Backend.Conf.t->(t,[>create_error])result(** Create a rw instance of [t] by creating the files.
Note on SWMR consistency: It is undefined for a reader to attempt an
opening before [create_rw] is over.
Note on crash consistency: Crashing during [create_rw] leaves the storage
in an undefined state.
Note on errors: If [create_rw] returns an error, the storage is left in an
undefined state and some file descriptors might not be closed. *)typeopen_rw_error:=[`Corrupted_control_fileofstring|`Corrupted_mapping_fileofstring|`Double_close|`Closed|`File_existsofstring|`Index_failureofstring|`Invalid_argument|`Invalid_layout|`Io_miscofIo.misc_error|`Migration_needed|`Migration_to_lower_not_allowed|`No_such_file_or_directoryofstring|`Not_a_directoryofstring|`Not_a_file|`No_tmp_path_provided|`Read_out_of_bounds|`Ro_not_allowed|`Sys_errorofstring|`V3_store_from_the_future|`Only_minimal_indexing_strategy_allowed|`Unknown_major_pack_versionofstring|`Index_failureofstring|`Sys_errorofstring|`Inconsistent_store|`Volume_missingofstring|`Multiple_empty_volumes|`Invalid_parent_directory|`Pending_flush]valopen_rw:Irmin.Backend.Conf.t->(t,[>open_rw_error])result(** Create a rw instance of [t] by opening existing files.
If the pack store has already been garbage collected, opening with a
non-minimal indexing strategy will return an error.
If [no_migrate = false] in the config, the store will undergo a major
version upgrade if necessary.
Note on SWMR consistency: It is undefined for a reader to attempt an
opening during an [open_rw], because of major version upgrades.
Note on crash consistency: If [open_rw] crashes during a major version
upgrade, the storage is left in an undefined state. Otherwise the storage
is unaffected.
Note on errors: If [open_rw] returns an error during a major version
upgrade, the storage is left in an undefined state. Otherwise the storage
is unaffected. Anyhow, some file descriptors might not be closed. *)typeopen_ro_error:=[`Corrupted_control_fileofstring|`Corrupted_mapping_fileofstring|`Io_miscofIo.misc_error|`Migration_needed|`No_such_file_or_directoryofstring|`Not_a_file|`Double_close|`Closed|`V3_store_from_the_future|`Index_failureofstring|`Unknown_major_pack_versionofstring|`Inconsistent_store|`Invalid_argument|`Read_out_of_bounds|`Invalid_layout|`Volume_missingofstring]valopen_ro:Irmin.Backend.Conf.t->(t,[>open_ro_error])result(** Create a ro instance of [t] by opening existing files.
Note on SWMR consistency: [open_ro] is supposed to work whichever the
state of the pack store and the writer, with 2 exceptions: 1. the files
must exist and [create_rw] must be over and 2. during a major version
upgrade of the files (which occurs during a [open_rw]).
Note on crash consistency: Crashes during [open_ro] cause no issues
because it doesn't mutate the storage.
Note on errors: The storage is never mutated. Some file descriptors might
not be closed. *)typeclose_error:=[`Double_close|`Index_failureofstring|`Io_miscofIo.misc_error|`Pending_flush|`Ro_not_allowed]valclose:t->(unit,[>close_error])result(** Close all the files.
This call fails if the append buffers are not in a flushed stated. This
situation will most likely never occur because the append buffers will
contain data only during the scope of a batch function. *)typeflush_error:=[`Index_failureofstring|`Io_miscofIo.misc_error|`Ro_not_allowed|`Closed|`Double_close|`File_existsofstring|`Sys_errorofstring|`No_tmp_path_provided]typeflush_stages:=[`After_dict|`After_suffix]type'ahook:='a->unitvalflush:?hook:flush_stageshook->t->(unit,[>flush_error])result(** Execute the flush routine. Note that this routine may be automatically
triggered when buffers are filled. *)typefsync_error:=flush_errorvalfsync:t->(unit,[>fsync_error])result(** [fsync] executes an fsync for all files of the file manager.
Note: This function exists primarily for operations like snapshot imports.
If fsync is enabled for the store (see {!Irmin_pack.Config.use_fsync}),
calls to {!flush} will also call fsync and therefore there is little need
to call this function directly. *)typereload_stages:=[`After_index|`After_control|`After_suffix]valreload:?hook:reload_stageshook->t->(unit,[>Errs.t])result(** Execute the reload routine.
Is a no-op if the control file did not change. *)valregister_dict_consumer:t->after_reload:(unit->(unit,Errs.t)result)->unitvalregister_prefix_consumer:t->after_reload:(unit->(unit,Errs.t)result)->unitvalregister_suffix_consumer:t->after_flush:(unit->unit)->unittypeversion_error:=[`Corrupted_control_fileofstring|`Corrupted_legacy_file|`Invalid_layout|`Io_miscofIo.misc_error|`No_such_file_or_directoryofstring|`Not_a_directoryofstring|`Unknown_major_pack_versionofstring]valversion:root:string->(Import.Version.t,[>version_error])result(** [version ~root] is the version of the pack stores at [root]. *)valcleanup:t->(unit,[>`Sys_errorofstring])result(** [cleanup t] performs cleanup operations for files related to GC. *)valswap:t->generation:int->mapping_size:int63->suffix_start_offset:int63->chunk_start_idx:int->chunk_num:int->suffix_dead_bytes:int63->latest_gc_target_offset:int63->volume:Lower.volume_identifieroption->(unit,[>Errs.t])result(** Swaps to using files from the GC [generation]. The values
[suffix_start_offset], [chunk_start_idx], [chunk_num], and
[suffix_dead_bytes] are used to properly load and read the suffix after a
GC. The value [volume] is used to reload the lower if it was modified by
the GC. The control file is also updated on disk. *)valreadonly:t->boolvalgeneration:t->intvalgc_allowed:t->boolvalsplit:t->(unit,[>Errs.t])resultvaladd_volume:t->(unit,[>Errs.t])resultvalgc_behaviour:t->[`Delete|`Archive](** Decides if the GC will delete or archive the garbage data, depending on
the presence of a lower layer. *)valgc_destination:t->[`Delete|`ArchiveofLower.t](** Returns where data discarded by the GC will end up. (see {!gc_behaviour}). *)valcreate_one_commit_store:t->Irmin.Backend.Conf.t->Control_file.Payload.Upper.Latest.gced->Index.keyPack_key.t->(unit,[>open_rw_error|close_error])result(** [create_one_commit_store t conf generation new_store_root key] is called
when creating a new store at [new_store_root] from the existing one,
containing only one commit, specified by the [key]. Ths new store will use
configuration options from [conf] and set to [generation]. *)endmoduletypeSigs=sigmoduletypeS=SmoduleMake(Io:Io.S)(Index:Pack_index.SwithmoduleIo=Io)(Errs:Io_errors.SwithmoduleIo=Io):SwithmoduleIo=IoandmoduleIndex=IndexandmoduleErrs=Errsend