Turns out we had a bad CSU/DSU on a T1 link recently, the thing that made it more difficult was that it was internal to the router (WIC-1DSU-T1)

Troubleshooting serial links is done by testing different parts of the links by looping endpoints.

The basic procedure:
http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a754b.shtml

The difference between loopback dte, loopback remote, and loopback line with pictures.

http://www.cisco.com/en/US/products/hw/routers/ps233/products_tech_note09186a0080093c56.shtml

How to reset the CSU:

http://www.cisco.com/en/US/products/sw/iosswrel/ps1826/products_configuration_guide_chapter09186a00800d9b62.html#xtocid22