AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: 5 Factors That Can Stretch Your Fine-Tuning Job on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

TL;DR

Hugging Face reports that a CatBoost model predicted the runtime of 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. The result is substantially more accurate than the tested FLOP-based estimate on this dataset, but it comes from one production environment and has not yet been validated in a live scheduling trial.

As detailed in the original analysis, Hugging Face reports that a CatBoost model predicted the end-to-end runtime of 272 supervised fine-tuning jobs with a mean absolute error of 1.71 hours in five-fold cross-validation. The result was far more accurate than the best tested FLOP-based estimate in the same dataset, but it comes from one production environment and does not yet show that using the model improves real-world GPU scheduling.

The jobs ran on shared NVIDIA H200 GPUs in a corporate production environment and accounted for about 7,382 GPU-hours. Their durations ranged from 1.6 to 172.4 hours. The dataset included model families up to 15 billion parameters, LoRA and full fine-tuning, varying dataset sizes, and jobs assigned one, two or four GPUs. The machines were drawn from a pool of 8 to 32 H200s across one to four nodes, with other training and inference work running alongside them.

Hugging Face defines runtime as the wall-clock period from when a job receives its GPUs until it finishes; queue time is excluded. The CatBoost model used 24 features, including configuration and model metadata available before dispatch, such as evaluation cadence, checkpoint settings, token volumes per GPU and fine-tuning method. A FLOP-based duration estimate was also included as one feature.

In the reported cross-validation, CatBoost had an R² of 0.98, a root mean squared error of 3.52 hours and a median absolute error of 0.51 hours, alongside the 1.71-hour mean absolute error. The FLOP-based baseline’s best tested setting, at 30% Model FLOPs Utilization (MFU), had a 12.41-hour mean absolute error. Its error was 13.01 hours at 45% MFU and 14.62 hours at 60%; the preview says a wider MFU sweep did not resolve the baseline’s shortcomings.

At a glance
reportWhen: Study preview reported; full paper fort…
The developmentHugging Face has shared preview results showing a CatBoost model predicted runtimes for 272 production fine-tuning jobs more accurately than the tested FLOP-based baseline.
AI Training · Briefing

Why Fine-Tuning Takes Longer Than FLOPs Say

Hugging Face tested runtime predictions on 272 production fine-tuning jobs. Configuration-aware estimates were more accurate in this dataset, but results from one production pool do not establish performance elsewhere.

1.71 hours
CatBoost mean absolute error
12.41 hours
Best tested FLOP baseline error
272
Production jobs evaluated
7,382
Approximate GPU-hours

What the estimates capture

The target is time from GPU allocation to job completion; queue time is excluded.

Compute

The training loop

FLOP estimates approximate compute using an assumed Model FLOPs Utilization, or MFU.

Configuration

The whole job

CatBoost uses job settings and model metadata, including evaluations, checkpointing and token volumes.

Shared systems

Other runtime costs

Setup, data handling, network and storage can add time alongside training.

Prediction error in the tested data

Lower error means predicted duration was closer to observed duration on average.

CatBoost1.71 hours
FLOP baseline, 30% MFU12.41 hours
FLOP baseline, 45% MFU13.01 hours
FLOP baseline, 60% MFU14.62 hours

What this could mean for GPU scheduling

More useful duration estimates could help teams plan scarce GPU capacity.

When jobs finish early
  • GPUs may sit idle until another job starts.
  • Operators can misjudge when capacity will be free.
When jobs run over
  • Other work can be delayed.
  • Delivery dates may shift.

How far the result reaches

The preview reports cross-validation on one corporate production environment.

1
Tested on one pool

Runs used shared NVIDIA H200 GPUs and varied configurations.

2
Generalization remains open

Accuracy on other clusters, software stacks and workload mixes is not established.

3
Operational impact remains unmeasured

The preview reports prediction error, not savings in GPU time or scheduling delays.

What to watchThe full paper is forthcoming; validation on independent runs and a live scheduler trial would show whether the result carries over.
The takeaway
Configuration-aware prediction beat the tested FLOP estimates on these jobs, but teams need broader validation before relying on the same accuracy for their own clusters.

More Accurate Runtime Estimates

Runtime estimates can affect how cluster operators allocate scarce GPUs and when they dispatch queued work. If a job finishes sooner than predicted, its GPUs may sit idle until another job starts. If it runs longer, it can hold resources past the expected finish and delay other work. More accurate estimates could help operators plan allocations and make schedules less vulnerable to these mismatches.

The reported gap suggests that a model using job metadata may capture factors a compute-only formula misses. But the study reports prediction accuracy, not operational savings. It does not measure whether a scheduler using CatBoost reduced GPU idle time, cost or delivery delays. Those outcomes remain unproven by the preview.

Bose QuietComfort Headphones (2nd Gen), Wireless Noise-Cancelling Over-Ear Headphones, White Smoke

Bose QuietComfort Headphones (2nd Gen), Wireless Noise-Cancelling Over-Ear Headphones, White Smoke

Ninja Foodi Smart 6-in-1 Indoor Grill and Air Fryer, 4 Qt, FG551

Ninja Foodi Smart 6-in-1 Indoor Grill and Air Fryer, 4 Qt, FG551

As an affiliate, we earn on qualifying purchases.

What FLOP Estimates Miss

A common estimate for dense Transformer training approximates compute as six times the parameter count and training-token count, then divides by GPU count, peak throughput and an assumed MFU. MFU is the share of a GPU’s theoretical peak throughput achieved in practice. The preview describes typical MFU as spanning roughly 30% to 60%, making the assumed value one source of uncertainty.

That calculation mainly estimates training-loop compute. A job’s measured wall time can also include setup, model loading, tokenization, evaluation and checkpointing, as well as uploads and final storage. Hugging Face says these overheads vary with configuration and can compete for network and storage capacity on shared clusters. The researchers included scheduled evaluation tokens and tested multiple MFU settings in their baseline comparison, but the preview’s central finding is still based on one production pool.

“Everything is known before dispatch.”

— Hugging Face, in its study preview

How Far the Results Travel

The reported scores come from five-fold cross-validation on 272 jobs from one corporate production environment. That evaluates predictions on partitions of the same dataset; it does not establish how the model performs on independently collected jobs from other periods, GPU types, software stacks or workload mixes. The preview does not report a live scheduling trial or a controlled measurement of GPU time, cost or delay saved.

The full paper has not yet been released, and the preview gives no publication date. Further details about the evaluation design and feature selection may be provided there. It is also unclear how often the predictor would need retraining as workloads and shared-cluster conditions change, or whether the reported error would hold under changing demand.

Full Paper and External Tests

Hugging Face says it plans to post the full version on arXiv, but has not given a date. The paper may offer more information about the dataset and evaluation method. Further tests on jobs collected from other time periods or computing environments would help establish whether the prediction accuracy carries beyond the production pool used here.

A subsequent live scheduler evaluation could test whether more accurate duration forecasts lead to fewer idle GPU intervals or delays. Until such results are reported, the preview supports a finding about prediction error on this dataset—not proof of broader deployment benefits.

Key Questions

What did the CatBoost model predict?

It predicted end-to-end wall-clock runtime for supervised fine-tuning jobs, measured from GPU allocation until completion. Queue time was not included.

How accurate was the prediction?

Hugging Face reports a 1.71-hour mean absolute error in five-fold cross-validation, with a 0.51-hour median absolute error. These are results on the study’s 272-job dataset, not a guarantee for other clusters.

How did it compare with a FLOP-based estimate?

The best tested FLOP-based setting, using an assumed 30% MFU, had a 12.41-hour mean absolute error. The comparison was conducted on the same dataset; the preview does not establish performance across independent sites.

Does the study show that GPU scheduling improved?

No. The preview reports runtime prediction metrics, not a live scheduling trial or measured reductions in GPU idle time, cost or delays.

When will the full paper be available?

Hugging Face says the full version will be posted on arXiv, but the preview does not give a publication date.

Primary source: Hugging Face · via ThorstenMeyerAI.com

You May Also Like

Top 8 Graphics Cards To Power AI Applications In 2026

Discover the leading graphics cards powering AI applications in 2026, including NVIDIA and AMD options, performance insights, and future-proofing features.

2026 Wireless Earbuds With AI: The Top 10 For Tech Enthusiasts

Discover the leading wireless earbuds with AI features in 2026, highlighting top models, their capabilities, and what makes them essential for tech lovers.

Game 3: Both Teams Destroy Inhibitors?

Both teams reportedly destroyed inhibitors during Game 3 of the LCS match, signaling aggressive play. Details remain unconfirmed and ongoing.

Black Ops 2 Ps5

Call of Duty: Black Ops 2 has been officially released for PlayStation 5, offering enhanced graphics and performance for players.