How much should I spend on a Protocol Analyzer?

This is the crux of the problem. Will an expensive analyzer deliver more than a cheaper one? Will I get more value from a higher cost product? My advice is to consider it very carefully before you decide. You can spend a significant amount of money on an analyzer, but you may not have to.

Proprietary solutions vary enormously in price and functionality. Although most make use of open formats (or at least allow data to be exchanged between different systems) you should check carefully that you are not tied into proprietary formats. It is very inconvenient to capture packets and then to have to mess around converting from one format to another if you need to share the information. Open Source analysis products have the huge advantage of being completely free, use open formats, and often provide as much functionality as proprietary solutions.

Decide on the features that you really need. If, in addition to protocol analysis, trending and performance measurements are very important to you a proprietary solution may be the best, since integration of the two functions is often very good. Again open source alternatives do exist so you could go for both a performance monitor and a protocol analyzer.

If technical support and training are important these are generally better provided for by proprietary solutions, though normally at additional cost.

If full wire speed packet capture is a requirement then you may have to consider a hardware solution, but these are extremely expensive and are normally only justified in special cases.

It is worth trying as many analyzers as possible to see which suits you best. For the types of problems described above the really important feature is the sophistication of the filtering mechanism. Again look carefully at what is being offered.