Step 1
By default, importing the library will import every component. If you are using webpack 1, then you can follow the directions shown here in the usage section for bundlers:
http://react.semantic-ui.com/usage#bundlers
The example configuration shows how you can use babel-plugin-lodash (despite the name) to automatically transform your import statements to individual component imports. Explicitly importing individual components will ensure that you are only bundling the components you are using. Unused components will not be included in your bundle.
Step 2
Every component includes a propTypes definition. These are only useful in development. They are also quite large and verbose. We wrap our prop type definitions so that they are automatically removed during minification with dead code elimination, provided that process.env.NODE_ENV is set to "production" and exposed as a global.
You should already be doing this as it is required by react to bundle in production mode. Just in case, here are the instructions on how to do this: How to turn on/off ReactJS 'development mode'?
Removing prop type definitions will save additional space.
Step 3
With the source code cut down to only include the components you are using, and those components cut down to production code only, you should now minify and compress your bundle.
Check the webpack docs for enabling production as this will minified your code and use dead code elimination, it is very straightforward. You can then compress your bundle with: https://github.com/webpack-contrib/compression-webpack-plugin.
Conclusion
There have been some updates to the library since I did this, but I achieved 81.8 kB for the entire library in UMD format, which has slightly larger overhead.
PR here showing the full setup: https://github.com/webpack-contrib/compression-webpack-plugin
Stats here: https://github.com/Semantic-Org/Semantic-UI-React/blob/3aa72bc76aac05c526e2b14a6ed4687995cd11af/stats-library.md