Google released the Gemini Distillation Service inside its Enterprise Agent Platform, first flagged on Hacker News in a thread that earned 17 points and 4 comments.
The service applies knowledge distillation to shrink Gemini models while preserving task performance.
What It Is and How It Works
Distillation trains a smaller student model to mimic a larger teacher model’s outputs. Gemini Distillation Service automates this pipeline for Gemini 1.5 Pro and Gemini 1.5 Flash teachers.
Users supply a dataset and target size; the service generates synthetic labels from the teacher, then trains the student on Google infrastructure.
Benchmarks and Reported Numbers
Early users report 3–5× latency reduction on classification and summarization tasks. One HN commenter measured a distilled 8B-parameter student reaching 94% of the teacher’s accuracy on a 5,000-example legal dataset while cutting inference cost from $0.0035 to $0.0007 per 1K tokens.
No official parameter counts or VRAM figures appear in the documentation yet.
How to Try It
Access requires a Google Cloud project with the Gemini Enterprise Agent Platform enabled.
- Navigate to the Models > Tuning > Distillation section in the console.
- Select teacher model (Gemini 1.5 Pro or Flash).
- Upload a JSONL dataset or connect a BigQuery table.
- Set student size (small, medium, or custom) and training steps.
- Launch; the service returns a new endpoint within the same project.
API calls use the standard Gemini tuning endpoint with distillation_config parameters.
Pros and Cons
-
Pros
- Native integration with Gemini data pipelines
- Automatic synthetic data generation
- Billing tied to training tokens only
-
Cons
- Limited student architecture choices
- No public benchmark leaderboard
- Requires Enterprise tier access
Alternatives and Comparisons
| Feature | Gemini Distillation | DistilBERT (Hugging Face) | OpenAI GPT-4o mini fine-tune |
|---|---|---|---|
| Teacher model | Gemini 1.5 series | BERT | GPT-4o |
| Max speed gain | 3–5× | 1.6× | 2–3× |
| Cost per 1K tokens | $0.0007 | Free (self-host) | $0.00015 |
| Dataset size needed | 1K–50K examples | 10K+ examples | 50+ examples |
| License | Google Cloud | Apache 2.0 | OpenAI terms |
Who Should Use This
Teams already inside Google Cloud running Gemini agents gain the most. Startups needing sub-100 ms latency on classification or extraction tasks benefit directly. Researchers wanting full control over student architecture or open weights should skip it and use open-source distillation libraries instead.
Bottom Line / Verdict
Gemini Distillation Service offers the fastest path for Google Cloud users to shrink Gemini models without leaving the platform, provided they accept the Enterprise tier and limited architecture options.
The low HN engagement suggests most practitioners are still waiting for public benchmarks before committing production workloads.
Top comments (0)