val create : filename:Base.string -> tcreate ~filename reads text file filename and stores its content in t. The tensors returned in this module use compact labels: e.g. if filename use n different characters, the values used in the tensors will range from 0 to n-1.
val iter :
?device:Device.t ->
t ->
f:(Base.int -> xs:Tensor.t -> ys:Tensor.t -> Base.unit) ->
seq_len:Base.int ->
batch_size:Base.int ->
Base.unititer t ~f ~seq_len ~batch_size iterates f over the whole dataset. f is given two different tensors xs and ys which shapes are both seq_len; batch_size; labels. ys is shifted by one compared to xs. The dataset is shuffled on each call to iter.
val char : t -> label:Base.int -> Base.charchar t ~label returns the character from the original file that has been mapped to label.
val total_length : t -> Base.intval labels : t -> Base.intlabels t returns the number of different labels, i.e. the number of distinct chars in the original file.