Hit mechanics differ between small arms and heavy weapons. Small arms normally are not able to penetrate an armored vehicle, but with a confirmed critical they are still able to deal damage to weak points like optics or night vision, reducing the gunner’s ability to spot and accuracy. A penetrating hit from a heavy weapon can have different results. A shot that remains intact will pass through crew and components in its way; if it reaches the other armored side then it may bounce and cause more damage, but if the penetration value exceeds the relative armor of both sides and the effective armor of crew and components, it will overpenetrate and pass clean through. The heavier the armor, the more likely a shot is to fragment through a dice roll, and this will result in damage in a cone pattern within the tank. If somehow a HE shell manages to penetrate the armor, then its explosion inflicts maximum damage to whatever is adjacent inside the tank and less damage to crew/components further away. Depending on encounter it’s more likely that HE explodes on the outside of the vehicle. This causes (reduced) sonic damage to crew within the tank and a die is rolled to determine whether spalling occurs (the larger the explosive, the more likely, and riveted armor is more likely to spall), which causes damage in a cone pattern but to a lesser extent than with an AP fragmentation. Explosions which don’t penetrate armor can still damage components like tracks.
Components that may be damaged include but are not limited to: engine, transmission, tracks, ammunition, main gun barrel, main gun breech, machine guns, turret ring, and fuel tanks. Damage to any of these reduces capabilities of the tank which should be obvious: in addition, critical damage to ammunition can cause an explosion which is the equivalent of a TPK, critical damage to engine or fuel can cause a fire (reduced chance if the vehicle runs on diesel), and a significantly damaged gun breech requires a roll to ensure it won’t explode upon firing. Although fire can be suppressed remotely (presumably) and characters with appropriate skills/feats may repair components inside the tank, to repair something outside like the tracks requires getting out of the vehicle.
A vehicle may have anywhere from three to five crew members: the most critical members are the driver, the gunner, and the commander, but a machine gunner and a loader are also likely to be present. Operation has its own skills, such as driving or gunnery, and even its own feats (like attacking with machine guns and main cannon at the same time); this makes fighting an experienced crew tougher. Some things may also require ability rolls such as strength for loading the gun and hand-turning the turret. The commander suffers a spot penalty while “buttoned-up,” but if he’s standing through his hatch enemies still have a hit penalty while targeting from front, sides, or rear (but not the top). A character on top of the tank may make a strength check to forcibly open a tank’s hatch while it’s occupied.
To keep the spirit of DnD it may be preferable to dismiss fixed values and instead use dice modifiers for armor class, gun penetration, and more. It’s a trade-off: the more dice are rolled per turn of battle the more variability exists but each round takes longer. The way I think it could work is this: an attacker rolls to hit a tank with an AP round and let’s assume a non-critical hit here. Penetration gets rolled with the modifier being the gun’s AP ability. The tank makes an opposed armor check with modifier depending on armor thickness and slope. If penetration wins out then damage is rolled. This makes a total of four rolls in every standard attack on a tank.
Because I felt like it I compiled a list of vehicle-mounted machine guns and cannons used by Britain, America, Germany and the USSR in WW2. It is by no means definitive or fully accurate but it gives a good overview of how weapons relate to each other in regards to calibre, barrel length, and penetration. Rather than including every single kind in a DnD home-brew it may be better and more flexible to come up with a formula that translates calibre and barrel length (measured in calibers) into penetration and damage modifiers in dice rolls. A wider/heavier shell and longer (proportional) barrel mean more force and therefore better penetration. A wider/heavier shell and shorter barrel (to be fair, from my table it looks like only the largest calibers benefit from a shorter barrel) allows for more explosive mass in HE shells, as shorter barrels exert less stress and allow for thinner-walled shells. Machine guns in DnD would operate differently and barrel length can be assumed to be irrelevant; RPM determines number of hit dice per round.
Forming categories of tank weapons by calibre may be helpful in simplifying this, though these are just suggestions. Smaller cannons suffer an accuracy penalty when hitting anything smaller than “large” while larger bores have a penalty hitting anything smaller than “huge”: