What does "having chemistry" actually mean?

I'm not talking about real life here.  Who cares about real life?  I'm talking about this phrase, "having chemistry" when it is used to describe the relationship between two actors and/or fictional characters.  I know what it is, I recognize it when I see it but what exactly is going on here?

Some of my favorite on screen couples are a great example of having fantastic chemistry - Scully and Mulder (The X Files), Sam Carter and Jack O'Neill (Stargate SG1), Veronica and Logan (Veronica Mars) and most recently Oliver and Felicity (Arrow) and Clarke with everyone (The 100).

In many cases, the "chemistry" takes the show runners by surprise.  I've read that Chris Carter abhorred the idea of Mulder and Scully being romantically involved but the tension between the  two characters just begged to go there.

The powers that be seemed to include some mild attraction between Sam and Jack from the very beginning but the show's loyalty to following realistic military protocol precluded going anywhere so they couldn't have suspected it would become the phenomenon that it was.  And that they'd be reviled for all time for never GIVING ANY CLOSURE OR PAYOFF.  Ahem.... Not that I'm bitter.

Felicity wasn't even a regular on Arrow to begin with and the show runners clearly intended Laurel to be Oliver's end game love as is the case in the comics the show is based on.  But Oliver and Laurel?  Boring and Irritating (with a capital I) right from the get go while folks shipped Oliver and Felicity as soon as they had exchanged three brief lines about "spilling" a "latte" (Oliver's explanation for why the bullet-riddled laptop he is giving her won't work) on a computer.  Their immediate "chemistry" changed the whole trajectory of the show.

I'm not sure what Rob Thomas had in mind with Veronica and Logan; if he intended Logan to become the character he became and be Veronica's one true love?  I suspect it was all part of his devilish plan - the man's a genius after all - but even though Logan was an epic asshat, the chemistry between he and Veronica was immediate.

Finally there's the situation with Clarke which is kind of new for me.  Usually I give my devotion to one ship but with Clarke, I'm all addlepated.  Clarke and Finn?  Hot! Clarke and Bellamy? Hot! Clarke and Lexa? Hot!  I'm beginning to think I'd 'ship the girl with a sofa cushion.  If they had sofa cushions in post-nuclear war America.

So all of these are very different situations and show but they are all defined by the chemistry shared by their characters/actors.  So what's going on here?  What creates this chemistry?   Here are some possible explanations I could think of:

1. The actors are good friends/really like each other: This doesn't work.  While Gillian Anderson and David Duchovny like each other, they famously had times where they were not getting along behind the scenes and Anderson was busy falling in love and getting pregnant by one of the crew members during the first season of the show.  However, their on screen chemistry never failed to be breathtaking.
2. They are just really good actors:  I'm not sure this is the whole picture either.  One couple that I didn't mention above was Buffy and Angel (Buffy the Vampire Slayer) who has immediate chemistry as well though, quite frankly, in the beginning, David Boreanaz couldn't act himself out of a paper bag.  With them maybe it was...
3. The couple represent a particularly attractive trope: Certainly a vampire slayer coupled with a vampire is intriguing, this doesn't particularly work for all of the above.  Logan is a serious jerkwad  - he's not just an adorably hot bad boy.  He is actually really terrible so how does that make him a hot match for V?
4. The actors have charisma:  The problem with this is that I have the same problem with defining charisma as I do "having chemistry".  It's a mystical intangible quality which is confusing.
5. Good character writing:  Now this has promise but also ... problems.  I do think this definitely plays a role.  I think all the characters above really work as representing interesting and real people.  However, why do I ship Clarke with everyone?  Is she just such a spectacularly written character that her attractive force is undeniable?  Maybe, but there may not be any character I love more than Veronica Mars.  She is SO well written but the only person for her is Logan.  Oh sure, she has a little chemistry with some other guys along the way but it only goes incandescent with Logan. 

What have I forgotten?  My list definitely feels incomplete.  I think if I was to make a guess I would say that "chemistry" is a combo of a few things just like the science of chemistry.  My vote would be a combination of good writing and good acting.  Having two characters that feel real in how they are written and two actors who play them naturally feels like it is what defines most of those mentioned above.  David Boreanaz sneaks in here and causes problem but I think in that case the situation is helped by the uniqueness and intrigue of the pairing and carries along until Mr. Boreanaz grows into his own.  And maybe you only need one of the actors to be really really good like Sarah Michelle Gellar.  But then you have some pretty good actors that have trouble finding romantic chemistry with anyone.  For example, the talented and beautiful Mr. Jensen Ackles who has WAY more chemistry with his brother Sam (Supernatural) than any lady they try to match him with.  Well, he and Castiel are also pretty adorable together.

If you want more of this adorableness you must immediately check out this list!

And then what about all those folks that don't "feel" the romantic chemistry? Or see the chemistry between a different pair than most others? What's up with the anti-shippers who think all this OTP and OTL stuff is a load of hogwash.  They think the 'shippers are reading into things and inventing and forcing a romance where none exists?  What's up with these folks - are they missing something or is seeing chemistry more a personal thing, that is really more about the viewers mindset and values and not about the actors/characters at all? Whoa.....  I think I just broke my brain.

What are your thoughts?  I would love to hear them because I am still very puzzled by the concept and am not convinced I know what I'm talking about.  Let me know!

P.S.  I realize that I only dealt with one kind of chemistry here - romantic chemistry - and that there are many other kinds of chemistry.  Exhibit A: Sam and Dean Winchester on Supernatural.