Fremont is about 20 miles away from Oakland and very close to San Jose. It'd be a problem for some A's fans to get to games there, but it'd be easier for others who are already close. The traffic and public transit things would need to be worked out but that's all to come.
Ideally, the A's would stay in Oakland. I'd love that. But the Coliseum was ruined for the casual fan when the Raiders came back. Mt. Davis is an eyesore and it changed the ambiance of the place. To some, along with past ownership and current, it's enough to keep them away.
Look at Candlestick. Did you go to a lot of games there? Was it a dump? Has the fanbase changed at all since the move into the new park? I think we know the answers to most of those questions.
Unfortunately, baseball in Oakland does not appear to be a high priority for Oakland itself. Jerry Brown was more focused on housing, not that he can be entirely blamed for that. The Coliseum location is a great one in terms of access to it, but the facility itself is outdated. Fans are too far from the action compared to other places. The concourse is too narrow. Views are blocked both there and in places like the bleachers. The field is still great, except when the Raiders start playing.
There are reasons why the Coliseum can still suffice for baseball but there are more reasons for the A's to get a baseball-only facility of their own. That is the direction MLB has gone in. The days of the multi-purpose stadium is over.