It was so damn confusing
My brother sat me down and drew me some sort of diagram
I think it goes like this (though I'm not brilliant at understanding this portion of the story)
Gregorovitch had the wand but it was stolen by Grindelwald. Dumbledore defeated Grindelwald and thereby obtaining the Elderwand. Decades later, Malfoy disarmes Dumbledore at the top of the tower, making him the rightful owner. However, Dumbledore was buried with the wand so Malfoy didn't really own the wand though he was the rightful owner.
Here is when it gets a bit confusing for me.
Harry disarms Draco at Malfoy Manner. Since Draco was the last rightful owner of the Elder wand, Harry (who disarms) him is now the new owner.
Voldemort decides to take the Elder wand from Dumbledore's grave but he doesn't understand that just to take doesn't make him the rightful owner because it wasn't obtained the proper way (through defeat or disarming, etc). During the final battle against Harry, he doesn't understand that Harry is the rightful owner and uses the Killing Curse.
Since the elder wand does not turn against it's master (Harry), Voldemort's curse rebounds.
I may be wrong on a few parts but the basic gist is there
So Gregorovitch -> Grindelwald -> Dumbledore -> Draco Malfoy - > Harry Potter