Other than the setting up email's and a public facing website, what else can I do with my domain name for my business?
Creating a website is pretty broad.
Have you considered starting a forum / community to attract your potential clients and have them engage in conversation about your market and products.
engage.yourdomain.com or community.yourdomain.com
You can create customer support systems by using tools like zendesk.com and whitelabeling it with your domain.
support.yourdomain.com or help.yourdomain.com
You can create customer satisfaction portals to communicate with and poll your customers directly about how they are using / interacting with your product.
satisfaction.yourdomain.com or telluswhatyouthink.yourdomain.com
In general any type of software base service you can provide to your customers that you think will enhance your business you can attach to your domain for easy access by your clients.
Depending on the suitability (length) of your domain a Short URL service can be quite catchy. You can use a site like http://domai.nr to find alternative domain suggestions, aka domain hacks, that can be quite eye catching.
If your company was called Get Wise for example you could register www.getwi.se which is the TLD for Sweden. You could then use YOURLS (http://yourls.org/ ) to create the same service that http://bit.ly offers by shortening long URLs.
It's great for marketing your site as a 'customised' domain like that can look good if done right and you can always make the service public to other people which could drive traffic to your site.