In　this　paper,　we　consider　the　optimal　approximations　of　univariate　functions　with　feed-forward　ReLU　neural　networks.　We　attempt　to　answer　the　following　questions.　For　given　function　and　network,　what　is　the　minimal　possible　approximation　error?　How　fast　does　the　optimal　approximation　error　decrease　with　network　size?　Is　optimal　approximation　attainable　by　current　network　training　techniques?　Theoretically,　we　introduce　necessary　and　sufficient　conditions　for　optimal　approximations　of　convex　functions.　We　give　lower　and　upper　bounds　of　optimal　approximation　errors,　and　approximation　rate　that　measures　how　fast　approximation　error　decreases　with　network　size.　ReLU　network　architectures　are　presented　to　generate　optimal　approximations.　We　then　propose　an　algorithm　to　compute　optimal　approximations　and　prove　its　convergence.　We　conduct　experiments　to　validate　its　effectiveness　and　compare　with　other　approaches.　We　also　demonstrate　that　the　theoretical　limit　of　approximation　errors　is　not　attained　by　ReLU　networks　trained　with　stochastic　gradient　descent　optimization,　which　indicates　that　the　expressive　power　of　ReLU　networks　has　not　been　exploited　to　its　full　potential.　(c)　2021　Elsevier　B.V.　All　rights　reserved.