If you want to get the best possible performance out of your site, then yes.
HTTP 304 ("Not Modified") results are certainly faster than uncached (HTTP 200) results, since the web server doesn't have to send any data to the browser, but the browser still has to make the request, so there's still a network round-trip involved. If the original result includes an HTTP Expires header, the browser doesn't need to contact the web server at all — it just uses data straight out of its cache.
There's a downside to using Expires headers, though: if you change files on your web site, the browser will never know it until the expiration date arrives. As such, Expires should be reserved for files which you don't expect to change on short notice.
Incidentally, Expires headers are a significant part of what makes this forum feel so fast. Pretty much every resource other than the main HTML pages are cacheable, so your browser can start rendering the page as soon as it receives data.