Overheard at the end of a race recently: "I don't know what happened, I do 150 mile rides all the time"
I don't care how much it impresses the people at your office, riding a kajillion miles is not that hard. You just sit on your bike and pedal all day. You don't have to do it particularly fast, and it never really hurts. Well not the way it hurts when you're getting dropped because you can't put out 400W for another 2 minutes. Racing a bike is not the same as riding a bike. Sure the physical motions may be the same, but they are entirely different worlds of pain. This is what separates us from the pathletes, the pretend tri-geeks, and the group riders. When the purpose of riding your bike is to race it, everything changes.
Most racing puts incredibly huge demands on your body for relatively short periods of time. An attack may last a few minutes, a hill may last half an hour, but that's pretty much it. Crits are even worse. You sprint out of every corner, gasp for air between them, and then sprint super duper hard one more time before it's all over. Going out and riding all day is not going to prepare you for this type of racing. Racing me on the path isn't going to either.
Learn to train, or just do whatever you want and be slow. Those are your options, but don't think that any racer is going to be impressed by your 500 mile week or your bike trip up the coast. Riding up lookout in 19 minutes is WAY more impressive than the 6 hours you rode yesterday. Try it sometime and you'll see why. Plus, I guarantee that anybody who can do a 20 minute lookout would beat you at a double century, they just have better things to do.