Story points are a good estimation technique that you can use when you try to estimate complexity in user stories. Unfortunately, complexity is not directly correlated to effort and time so you might end up having ultra complex features that can be implemented by a very inspired developer in a very short time (yes, miracles can happen from time to time).
On the other hand, you can have low complex features that require many hours of repetitive work.
Another variable in the equation is the QE time that you'll have to commit to test the feature. Again, there's no direct correlation between the implementation and testing complexity. For instance, automated testing can save a lot of man hours and make testing easier for a highly costing -from development perspective- feature.
A guess the caveat here is not try to use story points as what they are not: a metric for progress and effort.