Last year my son needed to glue some rubber washers together, and then to a metal washer. Same issue, contact didn't stick, neither did silicon which I was certain would work. After a bit of googling, it turned out superglue was the anwer.
I guess on a larger surface it might not be a solution...