Changes
2 changed files (+2/-2)
-
-
@@ -25,7 +25,7 @@ Since we'd like to use information from both sides of the gap when predicting the target distribution, this model is not autoregressive and can't be used for text generation.## Training I implemented this architecture using tch-rs (Rust bindings to PyTorch). The tricky part is that we need to mask the character that the model is trying to predict to prevent the model from cheating. My implementation applies two convolutions for the left and right of the target character and then adds them together. I used " " as a padding token (probably a bad idea). For more details, see my code at the end of this report. I implemented this architecture using tch-rs (Rust bindings to PyTorch). The tricky part is that we need to mask the character that the model is trying to predict to prevent the model from cheating. My implementation applies two convolutions for the left and right of the target character and then adds them together. I used " " as a padding token (probably a bad idea). For more details, see my code. I trained the model on WikiText-103 for 20 epochs using an AMD Radeon 7900XTX and fine-tuned for an additional epoch on the Gutenberg dataset. (Note to self: for training on ROCm, run using `LD_LIBRARY_PATH=.venv/lib/python3.13/site-packages/torch/lib:$LD_LIBRARY_PATH LIBTORCH_USE_PYTORCH=1 cargo r -r`.) It achieves a final loss of 0.37 which means it puts $e^{-0.37} \approx 0.69$ probability on the correct character on average.
-
@@ -34,7 +34,7 @@ ## Decoding For the no-breakpoint decoding, my algorithm first runs MCMC using bigram and trigram frequencies (computed from WikiText and Gutenberg) and then if the probability is high enough, it switches to MCMC using the language model. The bigram and trigram step is necessary to compute an almost-correct answer, since the language model gets confused when many of the characters are wrong. The language model computes for each character the distribution for which it should be replaced with, which I use to build a 28x28 matrix for how likely we should make each possible swap. My program then repeats this at most 50000 times using blazingly fast fearless concurrency and returns the answer with the lowest loss according to the language model. For the no-breakpoint decoding, my algorithm first runs MCMC for 10000 iterations using bigram and trigram frequencies (computed from WikiText and Gutenberg) and then if the probability is high enough, it switches to MCMC using the language model. The bigram and trigram step is necessary to compute an almost-correct answer, since the language model gets confused when many of the characters are wrong. The language model computes for each character the distribution for which it should be replaced with, which I use to build a 28x28 matrix for how likely we should make each possible swap. My program then repeats this at most 50000 times using blazingly fast fearless concurrency and returns the answer with the lowest loss according to the language model. To deal with breakpoints, my algorithm splits the input in half and decodes each half separately. If the first half was decoded correctly, it tries to decode the second half using the same permutation and uses the language model to detect when the text turns from coherent to gibberish. This narrows down the breakpoint location to within 20 characters, and then my algorithm brute-forces all those locations and picks the best one. Likewise, if the second half was decoded correctly, we repeat the same process. Sometimes it's possible for both halves to decode correctly or almost correctly if the breakpoint is near the middle.
-
-