Actually, I believe I may have figured it out when repopulating my lists.
When I started out I had 40+ items in my subcategory list and this was causing the function to do nothing at all. My first reaction was to change the columns from lookups to text - which while it did start filtering based on category, caused the error above.
After rebuilding the lists I only used a few records just to see if it would work, and it did; after which I repopulated the full lists and was back to where I started.
Since I noticed some discussions about the number of items in a list related to this call I decided to delete the list and go back to a few items, after which the filtering was working again but saving got me the ambiguous error again. I found out the reason for this is that after deleting the list you need to remove and put back the columns that look up to that list or you will get that error, even though it still filters properly.
So, two issues: One was not the column type but the number of items (combined with my jQuery and SPServices version apparently, based on some other discussions I'm searching through). The other was that deleting and recreating a list will break the lookups to that list.
I switched to 0.7.2 BETA and it fixed my first issue; the other I resolved just now.
Thanks again for this awesome library!