"Harvard has in recent years failed to live up to both the intellectual and civil rights conditions that justify federal investment," the letter reads.