I always thought of it as analogous to the silent Ls in would/should/could, calf, half, chalk, caulk, and Polk.