Just trying to make this a bit clearer - choosing Research all genes will research as many genes as you have vermillion. You know that researching 1 gene costs 100s and 1 Vermi. You have 20 genes overall in a pet (10 mother and 10 father), so researching them all costs 2000s and 20 Vermillion. If you don't have enough Vermillion on hand to research all 20 (or all unknown genes if you previously researched some), the game will randomly choose as many genes as you can afford.
Now the second qusetion though, Torhval is only half right. If you have the Inn (VERY useful building to have), you can invite a Geneticist, who will let you choose which genes to research. As a matter of fact, I almost always have a Geneticist at my Inn. It means I can check out mother and father of my pet and cross-reference which genes I am certain of, so I don't have to research those and can therefore save money (and especially vermi).