I've said this briefly before, but when two athletes of similar skill compete against each other, the one that is more conditioned/stronger will most likely win most of the time. Some people approach this by spending all of their time just trying to get more and more technical. Whereas others, either try to compensate greatly with strength/conditioning. Clearly at the highest level of BJJ, there is a good balance between the two. You will sometimes get an exception to the rule, but for the most part this applies.
So long story short: whether you are doing this seriously or casually, finding time to lift/condition yourself is always beneficial. Don't limit yourself!
Now a great match between JT Torres and Dan Simmler