My professor is trying to find a better format for his videos he makes he is currently using Windows Movie Maker to compress the videos and his first video is 60mb and he wants a format that takes up smaller size without much loss.

I have looked into http://www.matroska.org/index.html but I am having a bit of trouble converting them.

What software and format do you suggest he uses after he records his videos on his PC. I want to test it and show him the format and software on the already released videos first.

Its a tech class and he uses vmware to record his videos and uses a microphone to record his voice.

Any help would be greatly appreciated.

I can't say I've ever used Matroska, but I do know that the WMV format is very good for maintaining a high quality/small file size ratio and there won't be many that beat it.

You could give DivX a try (free trial available and then very cheap to purchase). You could also give the Quicktime Codec a try, certainly does produce high quality results but I don't think a trial of Quicktime Pro is provided.