(This piece originally appeared in the “Will we ever?” column at BBC Future. Though I added a few visuals, I left the UK spellings and BBC stylebook conventions in place. My thanks to Simon Frantz for editing the text from my usual ramblings into something coherent. Even if he did cut out my Pink Floyd song quote.)

Beautiful as it is, the Universe hides a great deal of its secrets from us. All the stars, galaxies, and other objects we see mask the presence of another substance that comprises 84% of all the mass in the cosmos. That substance, which in our ignorance we call dark matter, has proven annoyingly elusive for the better part of a century.

Detecting dark matter directly has proved to be tricky, but there is strong evidence for its existence, and it comes from a variety of sources. The first hint of dark matter’s existence came in 1933, when Swiss astronomer Fritz Zwicky calculated that the Virgo Cluster of galaxies didn’t have enough visible matter in the form of gas and stars to hold it together. It must be embedded in a halo of something invisible, or else it would fly apart. In the early 1970s, American astronomer Vera Rubin demonstrated the magnitude of the missing mass problem when she showed that the outer regions of spiral galaxies rotated far too quickly unless the galaxies contained a lot more mass than could be seen.

In the intervening decades, the evidence for dark matter has only grown stronger. Galaxy clusters—including the iconic “Bullet Cluster” discovered in 2006 – clearly contain far more mass than is visible in stars and gas. On even larger scales, galaxies collect into clumps and long filaments, rather than falling into random configurations, and surround regions largely devoid of matter.

So if dark matter does exist, what is it made of? Right now, a far easier question to answer is what dark matter isn’t. First of all, the name is misleading: dark matter isn’t “dark” in any usual sense of the word. “Invisible matter” is a better term: light shining on dark matter from any source passes right through without being absorbed or scattered, regardless of the type of light. This means dark matter can’t be made of atoms or of their constituent parts; that is, electrons, protons and neutrons.

In fact, dark matter doesn’t correspond to anything in the Standard Model, the best explanation we have for how the universe works. The Standard Model describes many aspects of ordinary matter as we know it, along with three of the four fundamental forces: the electromagnetic force, the weak force, and the strong force. (The fourth force, gravitation, lies beyond the Standard Model, it is governed by Einstein’s general theory of relativity.) It also includes a lot of objects that exist ephemerally –including particles made of exotic quarks that decay very quickly – or which are common but hard to detect, such as neutrinos. The final piece of the Standard Model may have been found last summer, with the discovery of something resembling the Higgs boson at Cern.

Cold evidence

Dark matter is none of the above. In particle physics terms, dark matter doesn’t interact with the electromagnetic force, which governs light, and it ignores the strong force, which binds atomic nuclei together. But understanding how it behaves gravitationally does offer some clues to its make-up.

The way in which galaxies collect into clumps and long filaments makes most sense if the Universe began as a nearly uniform cloud of dark matter, but with tiny fluctuations in density, where a little more dark matter than average would sit. That over-density would attract more matter, creating lumps in some places and emptying out other regions. And thanks to people measuring the faint echoes of the birth of the universe – the cosmic microwave background – we know how much matter is made of atoms, and how much is invisible.

From these data, we can learn a few important things. First, dark matter seems to act like particles, based on the way it collects and behaves under the influence of gravity. Also, dark matter is “cold”, meaning that it doesn’t move very rapidly; dark matter that is too warm will resist clumping together as much as we observe. That in turn places some limits on the lowest mass dark matter particles could have: if they’re too light, they tend to move faster. Neutrinos, despite being extremely numerous and not affected by the electromagnetic force, are too light and therefore too “hot” to be dark matter.

To keep things cold, dark matter particles must not collide readily, either with each other or with ordinary matter. That’s because collisions transfer energy, leading to higher speeds, and ultimately in warming up the dark matter. A lot of research is focused on testing how much dark matter can collide in particularly dense regions, such as near the centres of galaxies.

Diary of a WIMPy particle

Observational evidence may have ruled out two of the three non-gravitational ways that dark matter could interact with ordinary matter, but that still leaves one possibility: the weak force. In fact, several extensions to the Standard Model predict the existence of weakly-interacting particles, including a popular theory called supersymmetry (Susy, often pronounced SOOsee) – one of the cornerstones of string theory. The most promising dark matter candidate that encompasses all these possibilities is known as a Wimp: a weakly-interacting massive particle.

However, that’s a big category. Worse still, models don’t predict a specific mass for any Wimp candidate, leaving experimental physicists with ranges of possible values. (That’s not a problem specific to Wimps, of course: the Higgs boson also lacked a specific mass prediction from theory.)

Despite all these issues, a variety of experiments and observations have helped trim down the Wimp mass range.

For example, the XENON100 experiment, located deep under the Gran Sasso mountain in Italy, contains a target of liquid xenon at -91C in a stainless steel cylinder. As with neutrino detectors, putting dark matter detectors underground helps prevent cosmic ray particles from interfering with the equipment. If a Wimp enters the cylinder and hits a xenon atom, the atom responds by losing electrons. Electrically charged plates at the ends of the cylinder push the electrons to the top, where they are recorded, along with any light the xenon gave off as a result of the collision. Because they interact so weakly with matter, each Wimp is unlikely to collide with anything, so the XENON100 team looks for single collisions. Anything more than that, and the collisions were probably another type of particle, such as a neutron.

In July 2012, the XENON100 research team announced that they had failed to find any Wimp signature above a certain mass limit, after operating for 225 days spread over 13 months. In particular, these data excluded several possible Wimp ranges from Susy, which is consistent with other disappointing results from particle colliders. The XENON100 results also rule out possible detections from other experiments, leaving dark-matter hunters gnashing their teeth.

However, the major dark matter detection experiments are upgrading, and new ones are in the works. The major goals in all of these efforts include reducing background signals (from cosmic rays and other particles) and increasing the sensitivity in mass ranges where various models predict Wimps may be hiding.

Exotic forces

If particle detectors cannot unmask dark matter’s identity, maybe astronomy can come to the rescue. Just as ordinary matter comes with antimatter partners, dark matter should have its counterpart. So detecting annihilation of dark matter/antimatter pairs in dense regions in space could offer some indirect evidence. Data from the orbiting Fermi Gamma Ray Observatory showed a tantalising signal coming from near the Milky Way’s centre that could possibly have been from Wimp annihilation. However, the result dwindled into insignificance after further analysis. Similar observations of dwarf galaxies also failed to find clear dark-matter annihilation signatures.

Of course, dark matter may not be Wimp-y. If it’s not, one intriguing possibility is that the four forces of the Standard Model aren’t the only way dark matter could interact with ordinary matter. The weak force is much weaker than the electromagnetic force, but there could be another so-called superweak force, which is negligible under most circumstances. One unpleasant possibility is that dark matter doesn’t interact with ordinary matter at all, except through gravity. If that’s the case, as with Wimps our best hope may be looking for dark matter annihilation. The Alpha Magnetic Spectrometer (AMS) on the International Space Station is currently surveying the skies for high-energy particles, or cosmic rays, and researchers have hinted that they have a big dark matter announcement coming in the next few weeks.

While a positive detection from AMS likely won’t reveal the full identity of dark matter, it might help us determine whether our Wimp searches are doomed to failure or not. In the end, the Universe may demand a high level of cleverness from us if we are to have any hope of identifying what dark matter is, as opposed just to what it does. But if the search for the Higgs boson has taught us anything, it’s that we relish the challenge of detecting the seemingly undetectable, even if it takes decades to succeed.