I have been working on a startup for 5 months. The startup has some outside investment and money in the bank. My co-founder is CEO and never set up payroll. Instead, he had us taking money directly out of the business account with the intent of rolling this over into payroll. During this time, I did not take much at all from the business and actually went into debt because I believed we would raise money. When I came to the conclusion that my co-founder was not really working and was unreliable, I left. But before leaving, I took money out of the business to compensate me for what I would have been paid had he set up payroll. We never explicitly defined in writing what this pay would be and now he is trying to say that it would have been much lower than I ever would have agreed to and he is demanding I pay it back. Since there was no written agreement, am I in any way legally responsible because I took this money directly out of the business account and not through payroll?
I would suggest you get a lawyer. It sounds like you had some embezzlement going on in your company. Although everyone seems to have participated, it is embezzlement nonetheless. I also suspect that payroll taxes were not paid. Big problem. If this explodes - you'd better be well covered. Do not discuss this with anyone unless client-attorney privilege applies. Anonymous Internet forums included, you'd be surprised how easy it will be to find this post once one (the outside investor?) knows what to look for, and tie it back to you.
Get a lawyer. Fast.
You use to have access to the accounts, so there must have been some agreement made that allowed you to take money. Now you've left the company and have no control over how they are going to do the accounting, payroll etc.
Right or wrong, they could declare you as a contract worker. You will have to handle your personal taxes a little differently if this is the case.
The entity that funded the company may have provisions in the agreement with your company on what could and could not be done with the money. While you were a partner, you may have broken a covenant in this agreement. At the least they have an expectation of following sound accounting practices.
Get a lawyer and hope you can come to some agreement.