Accuracy-based evaluation cannot reliably distinguish genuine generalization from shortcuts like memorization, leakage, or brittle heuristics, especially in small-data regimes, according to https://arxiv.org/abs/2603.23517. The position paper argues for mechanism-aware evaluation combining task-relevant symbolic rules with mechanistic interpretability to produce algorithmic pass/fail scores showing where models generalize versus exploit patterns. In an NL-to-SQL demonstration, two identical architectures were trained under different conditions; the model without schema information reached 94% field-name accuracy on unseen data via standard metrics but violated core schema generalization rules under symbolic-mechanistic evaluation.