Ablative armor is used on some Federation starships such as USS Defiant and USS Prometheus to reduce "bleedthrought" damage and protect ship from enemy weapons fire in case shields are disabled.
From name "ablative" we can assume that it works by vaporizing/reducing itself into powder when hit by beam weapon. However, visual effects seem to suggest that in addition to ablation effect, it also dissipates energy of beam over large area, thus further reducing damage to hull. It also means it is more efeective against energy weapons than photon torpedoes, notion which is supported in several DS9 episodes, especially episode "Starship Down", where torpedo is able to penetrate Defiant's hull without much trouble.