For the absolute cheapest way of starting a website. As well as having the most control, and best performance.
Put your website on a desktop you are not using. Make it your web server. Even if it is 8 years old - You will get better performance (in terms of running scripts, multiple users, etc) than a shared hosting plan.
If you dont know how to do a Webserver you can always download Xampp for windows. Only thing you need to do, is put the server's IP address outside the router (100's of tutorials on the web), and leave the box on. Plus some minor configuration changes to Xampp for security reasons, but it shows you how. Plus whoever you buy the domain from, you point to the server's IP
In my opinion that is the best way.