Large language models (LLMs) struggle with reliable rule-based reasoning, particularly in mathematical domains requiring precise computation. Recent neurosymbolic approaches have shown promise by encoding LLM hidden states into neurosymbolic representations to perform rule-based reasoning, but current methods rely on linear encoders applied to a single token’s hidden state, limiting representational capacity and robustness to input variations. This work extends previous neurosymbolic LLM methods by replacing linear encoders with transformer-based architectures that encode multiple hidden states from the model’s forward pass, enabling the model to capture information distributed across longer input sequences. We evaluate the method on arithmetic prompts in a low-data 3-digit setting with varied question formats and in 4/6-digit scaling experiments. The transformer encoder achieves 93.1 percent average accuracy, compared to 53.6 percent for the linear encoder, and remains substantially more robust as input length increases. We make our code available at https://github.com/vdhanraj/Neurosymbolic-LLM/tree/transformer_encoder.