The fastest I got it was 114 tok/s using 8 concurrent calls at the same time, and a little over 30 tok/s for single stream generation. It uses my "shortlist mtp" modification that only allows mtp to use 16,000 tokens instead of the full dictionary, but that speeds things up even though it causes slightly lower draft acceptance percentages.
Published an update to https://GrumpyDev.ai today. Lots of simplification to remove some cruft, and some site updates to make the wording more clear. If you do a lot of AI coding, you should check it out.
While doing various projects I kept running into situations where I wanted to be able to have representative samples of some of the current large SOTA datasets that were smaller so I didn't need to worry about slicing or anything else at runtime. So, I created sub datasets making sure to keep the same ratios of data sources. Each dataset card provides info for what's in it. 100M token datasets: RedPajama v2 100M Falcon RefinedWeb 100M Cosmopedia 100M
