Recently I had a problem where one of my script would insert duplicates in a table in certain situations. So I had to de-depulicate my entries.

I found a post that solved this issue but it didn't quite satisfy me so here's my solution :

Create a new table that is a copy of the original one and add a unique index on whatever cols form something that is indeed unique (for me it wasn't quite as simple as putting a unique index on an id field).

Now the only thing you need to do is insert the content of the old table in the new table while ignoring duplicates. This is achieved with the following line (also I wanted to remove anything older than january first of this year) :

INSERT IGNORE INTO new_table SELECT * from old_table WHERE date >= '2009-01-01 00:00:00';

Now you can either drop the old table or move it out of the way and replace it by the new one by doing a rename.

 RENAME TABLE new_table TO old_table;