Thirdly, the rebel fleet was meant to be much bigger than it actually looks - apparently Lucas wanted a bigger variety of ships but couldn't get enough models together in time... So the rebel fleet looks tiny, but is meant to be large, so it is smaller than the Imp fleet and not a match in principle, but has a fighting chance. Both the fighters and the larger ships were more numerous than seen on screen at any time, and it's never suggested that any shot of the rebel fleet shows the ENTIRE fleet. The number of starship engines seen from the rear when Wedge and then the Falcon escape the Death Star is quite numerous, and this is AFTER the various losses in the battle - suggesting an initially very large fleet.
If Red, Gold, Grey and Green groups are the entire rebel fighter force, then the total number of fighters is only 48 - clearly outnumbered and overwhelmed by the TIES. However, each "group" may have been a wing of three or four squadrons. Or the ships which came out of hyperspace may have been those intended to go into the Death Star core, while further snubfighters were transported on the larger ships and deployed later. (Otherwise, it is anomalous that the Rebels would waste fuel sending snubfighters through hyperspace, risking their running out of fuel if there's a long battle, when they could be carried on the larger ships and deployed at the battleground).
It is anyway widely established that rebel fighters are better than TIEs in many ways (especially in being shielded), and it's usually suggested that a Mon Calamari Star Destroyer is roughly equivalent in fighting power to a Star Destroyer.
None of the rebel ships engage with Star Destroyers until they move into very close range - their role was to "keep [the rebels] from escaping" (Admiral Piett), so presumably the Star Destroyers were either blocking the main escape route so the rebels couldn't simply flee, or were shielding Interdictor cruisers which were keeping the Rebels from going to hyperspace (though I've got problems with whether Interdictors - a non-Lucas invention - really make sense in the context of the films). Meanwhile, the TIEs were keeping the rebels occupied through the classic imperial tactic of attacking in waves.
Also, only snippets of the battle are actually seen in the film... Some ships which appear (e.g. B-wings) are never actually seen in combat. B-wings are heavy attack bombers which can take out a Star Destroyer bridge tower and are shown doing so in a famous picture which was released at the time of the films and shown on the B-wing toy's box. Star Destroyers, like many Imperial vehicles and vessels (e.g. Death Star, AT-ATs) seem to have an achilles' heel - for Death Stars the exhaust port and reactor core, for AT-ATs the legs (and other places such as the underside hit by Luke's grenade), for Star Destroyers the bridge and the main shield generators on top of it. Take out the bridge and the entire Star Destroyer will either blow up or be incapacitated (as if shown several times, most notably with the Super Star Destroyer Executor).
Similar manoeuvres and rebel victories, based on tactical skill, greater intelligence and acumen, and/or notorious Imperial weak spots, crop up throughout the "secondary" novels and comics (especially the X-Wing series), making the Endor victory seem less anomalous. One could take as examples Rogue Squadron's defeat of the Lancer Frigate in X-Wing Book 1, the defeat of the Lusankya and accompanying Star Destroyer in Book 4, and the victory at the site of the Katana Fleet in Zahn's The Last Command.
