WWF caught two images of the animal, which is bigger than a domestic cat, dark red, and has a long muscular tail.
Local people, the WWF says, had not seen the species before, and researchers say it looks to be new.
The WWF says there is an urgent need to conserve forests in south-east Asia which are under pressure from logging and the palm oil trade.
The creature, believed to be carnivorous, was spotted in the Kayan Mentarang National Park, which lies in Indonesian territory on Borneo.
The team which discovered it, led by biologist Stephan Wulffraat, is publishing full details in a new book on Borneo and its wildlife.
'You don't find new mammals that often, and to do so must be extraordinary,' said Callum Rankine, head of the species programme at WWF-UK.
'We've got camera traps there, which are passive devices relying on infra-red beams across forest paths,' he told the BBC News website.
'Lots of animals come past - it's much easier than pushing through the forest itself - and when an animal cuts the beam, two cameras catch images from the front and back.'