I don't know which size loom you are using - but I use the lime green knifty knitter loom for adult hats (with and without brims) - and that works fine - don't really bother with inches though!
As far as your colors, the first stitches you put on the loom usually are the ones that are by your face - and then you will gather the finished tube together at the top, (last stitches on the loom) which is the top center (on top of your head, or closed end back of hat -
Remember, if you want an extra edging/rim around your hat, (by your face) you will probably be looming that first, and then folding it over on the pegs, before continuing -