HTML blog me nyatakaka bliboae nye esi:

Fia HN: Kpɔɖeŋu Hehena Ŋkuɖodzi Simulator — Nusita GPU Ŋkuɖodzi Ðoɖowɔwɔ Le Vevie Wu Ɣesiaɣi

GPU ƒe ŋkuɖodzinu ƒe hiahiãwo ƒe akɔntabubu hafi adze kpɔɖeŋu hehenana ƒe duɖimekeke gɔme nye mɔxenu siwo dzi woŋe aɖaba ƒui wu gake woxɔa ga geɖe le mɔ̃ɖaŋununya sɔsrɔ̃ ƒe dɔwɔwɔwo me la dometɔ ɖeka. Model Training Memory Simulator yeye si le ʋuʋu ɖi, si woɖe fia nyitsɔ laa le Hacker News, kpɔa kuxi sia gbɔ ŋkume kple ŋkume to mɔɖeɖe na mɔ̃ɖaŋudɔwɔlawo be woagblɔ VRAM zazã ɖi, ade dzesi ŋkuɖodzinu ƒe mɔxenuwo, eye woawɔ hehenana ƒe ɖoɖowo wòanyo wu — esiawo katã hafi tensor ɖeka nadze GPU dzi.

Nukae Nye Kpɔɖeŋu Hehena Memory Simulator Kple Nukatae Wòle Be Nàtsɔ Ðe Le Eme?

Kpɔɖeŋu hehenana ŋkuɖodzinu simulator nye dɔwɔnu si bua GPU ŋkuɖodzinu ƒe afɔɖoƒe si wokpɔ mɔ na le nusɔsrɔ̃ deto hehenana dɔ me si wotu ɖe kpɔɖeŋu ƒe xɔtuɖaŋu, batch ƒe lolome, precision format, optimizer tiatia, kple parallelism strategy dzi. Le esi teƒe be woatrɔ alilikpo xɔasiwo ƒe kpɔɖeŋuwo ko be woado go CUDA Out of Memory vodada siwo wovɔ̃na na aɖabaƒoƒo ʋɛ aɖewo le hehexɔxɔ me la, mɔ̃ɖaŋudɔwɔlawo ateŋu awɔ ŋkuɖodzinu ƒe nɔnɔmetata bliboa ƒe kpɔɖeŋu do ŋgɔ.

Fia HN dɔa tsɔa mɔ si le ʋuʋu ɖi kpɔa kuxi sia gbɔ, si naa mɔnu bubu si me kɔ, si nutoa me tɔwo ʋãna ɖe dɔwɔnu siwo nye ame ŋutɔ tɔ ƒe nɔnɔmetatawo wɔwɔ teƒe. Ebua akɔnta le parameters, gradients, optimizer nɔnɔmewo, activations, kple framework overhead — atɔ̃ vevi siwo kpena ɖe GPU ŋkuɖodzinu zazã le hehexɔxɔ me. Le ƒuƒoƒo siwo le dɔwo wɔm le NVIDIA A100s, H100s, alo RTX kaɖi siwo sɔ na nuƒlelawo gɔ̃ hã dzi gome la, ɖoɖowɔwɔ do ŋgɔ sia ƒomevi ateŋu adzra dɔlar akpe geɖe ɖo le akɔntabubu gbegblẽ kple gaƒoƒo geɖe ƒe ɣeyiɣi si woatsɔ aɖɔ vodadawo ɖo me.

Aleke GPU Memory Zãna Le Model Hehenaɣi?

Afisi ŋkuɖodzinu yina le hehexɔxɔ me gɔmesese le vevie ŋutɔ na ML mɔ̃ɖaŋudɔwɔla ɖesiaɖe. Simulator la ma nuzazã ɖe hatsotso vovovo siwo woate ŋu agblɔ ɖi me:

ƒe nyawo
• Kpɔɖeŋu ƒe Nɔnɔmewo: Ahɔhɔ̃mekawo ƒe kpekpeme xoxowo. 7B-parameter model le FP32 me zãa abe 28 GB ene na kpekpeme ɖeɖeko, eye wòɖiɖina va ɖoa 14 GB le FP16 alo BF16.
• Gradients: Wodzrae ɖo le backpropagation me, gradients zi geɖe la, woɖea ŋkuɖodzinu ƒe afɔɖoƒe si le parameters ŋutɔ me fiana.
• Optimizer States: Adam kple AdamW léa nɔnɔme tensor eve bubuwo ɖe asi le parameter ɖesiaɖe me (ɣeyiɣi gbãtɔ kple evelia), le nyateƒe me la, edzia parameter ƒe ŋkuɖodzinu zi gbɔ zi etɔ̃ ne wole FP32 optimizer nɔnɔmewo zãm.
• Dɔwɔwɔwo: Wodzra domedomenuwo ɖo hena megbedede. Esiawo dzidzena kple batch ƒe lolome kple ɖoɖo ƒe didime, si wɔe be wonye esiwo trɔna wu — eye zi geɖe la, wonye gãtɔ kekeake — ŋkuɖodzinu zãla.
• Framework Overhead: CUDA ƒe nɔnɔme, ŋkuɖodzinu ƒe mama, kadodo ƒe buffers na hehenana si woma, kple ɣeyiɣi kpui aɖe ƒe mama siwo gbɔgblɔ ɖi sesẽ ne womewɔe abe alesi wowɔe ene o.

ƒe nyawo

ƒe nyawo

Key Insight: Le gbegbɔgblɔ ƒe kpɔɖeŋu hehexɔxɔ ƒe duƒuƒu gã akpa gãtɔ gome la, optimizer nɔnɔmewo kple activations — menye model weights ŋutɔ o — ye nye ŋkuɖodzinu zãla vevitɔwo. Memory simulator ɖea gbagbã sia fiana hafi tsɔa ɖokuiwò nana ɖe hardware xɔasiwo ŋu, si trɔa akɔntabubuwo wòzua mɔ̃ɖaŋudɔwɔwɔ.

ƒe nyawo

Nukae Na Open-Source Simulator Sia To Vovo Tso Dɔwɔnu Siwo Li Gbɔ?

Hacker News ƒe habɔbɔa wɔ nu ɖe dɔ sia ŋu elabena eƒo nu tso vevesese ŋutɔŋutɔ siwo gbɔkpɔnu siwo li fifia gblẽ ɖi womakpɔ wo gbɔ o ŋu. Alilikpo me dɔwɔƒe akpa gãtɔ naa GPU ŋkuɖodzinu ƒe akɔntabubumɔ̃ veviwo, gake ƒã hafi wobua akɔnta le hehenanamɔnu siwo me wowɔa nu pɛpɛpɛ le tsakatsaka me, gradient checkpointing, tensor parallelism, alo ZeRO-stage optimizations tso frameworks abe DeepSpeed kple FSDP.

Simulator sia wɔa ɖoɖo deŋgɔ mawo ƒe kpɔɖeŋu tẽ. Mɔ̃ɖaŋudɔwɔlawo ateŋu ade woƒe ɖoɖo tɔxɛ — gblɔ be, 13B ƒe kpɔɖeŋu si me ZeRO Stage 3 le, gradient checkpointing si wowɔ dɔ, BF16 mixed precision, kple micro-batch ƒe lolome si nye 4 le 8 GPUs me — eye woaxɔ ŋkuɖodzinu ƒe gbagbã tsitotsito le mɔ̃ ɖesiaɖe dzi. Nu tɔxɛ ƒe seƒe mae nye nusi ma ɖoɖowɔwɔ ƒe dɔwɔnu si ŋu viɖe le tso akɔntabubu si le megbe le agbalẽkotokua me gbɔ.

Nɔnɔme si le ʋuʋu ɖi hã fia be nutoa me tɔwo ateŋu akekee ɖe enu. Woateŋu atsɔ xɔtuɖaŋu tɔxɛwo, optimizer ƒe dɔwɔwɔ yeyewo, kple hardware profile siwo le dodom la katã akpe asi ɖe eŋu, si ana dɔwɔnua nasɔ esime ML ƒe nɔnɔme le tɔtrɔm le breakneck speed me.

Aleke Asitsahawo Ate Ŋu Akpɔ Viɖe Tso Dɔwɔɖoɖo Siwo Me Nunya Le Wu Me?

Togbɔ be wotu simulator la na ML mɔ̃ɖaŋudɔwɔlawo hã la, gɔmeseseawo keke ta ɖo habɔbɔ ɖesiaɖe si de ga AI ƒe ŋutetewo me. GPU ƒe kpɔɖeŋuwo ƒe ɖoɖowɔwɔ fũ akpa le ŋkuɖodzinu ƒe nudidi siwo ŋu kakaɖedzi mele o ta naa alilikpo me fewo dzina ɖe edzi. Nusiwo womewɔna nyuie o hea hehenana ƒe duƒuƒu siwo do kpo nu, mɔ̃ɖaŋudɔwɔwɔ ƒe gaƒoƒo gbegblẽ, kple kpɔɖeŋu ƒe dɔwɔwɔ siwo tsi megbe.

Na asitsaha siwo le dzidzim ɖe edzi siwo le dɔwɔwɔ ƒe dɔwɔwɔ geɖewo dzi kpɔm — tso dɔa dzikpɔkpɔ dzi va ɖo gazazã ŋuti ɖoɖowɔwɔ dzi va ɖo asisiwo me dzodzro dzi — gɔmeɖosea sɔ: wɔ kpɔɖeŋu hafi nàtsɔ nunɔamesiwo ade asi. Eɖanye GPU ƒuƒoƒowoe nèle ɖoɖo wɔm alo nèle asitsadɔwɔnu siwo nàwɔ dɔ na wò ƒuƒoƒoa tiam o, nunɔamesiwo ƒe hiahiãwo ƒe nɔnɔmetata si me kɔ hafi nàdzidzee xea mɔ na gbegblẽ eye wònaa emetsonuwo wɔa kabakaba.

| Susu si nye be woawɔ nunɔamesiwo ƒe hiahiãwo ƒe kpɔɖeŋu hafi woatsɔ wo ade dɔwɔwɔ me la ku ɖe asitsatsa ƒe dɔwɔnawo ŋu ŋusẽtɔe abe alesi wòku ɖe hehenana ƒe kpɔɖeŋu ŋu ene.

Nyabiase Siwo Wobiana Enuenu

Ðe ŋkuɖodzinu ƒe nɔnɔmetata ateŋu axe mɔ ɖe vodada siwo le ŋkuɖodzinu godo nu keŋkeŋ le hehexɔxɔ mea?

Simulator ɖea afɔkua dzi kpɔtɔna ŋutɔ to akɔntabubu siwo sɔ si wotu ɖe wò ɖoɖowɔwɔ dzi nana me, gake mateŋu abu akɔnta le dɔwɔwɔ ƒe tɔtrɔ ɖesiaɖe ŋu o. Akɔntabubu ƒe nɔnɔmetata siwo le ŋusẽ kpɔm, nyatakaka siwo ƒe didime trɔna, kple ame etɔ̃lia ƒe agbalẽdzraɖoƒe ƒe ŋkuɖodzinu ƒe sisi ate ŋu ahe gazazã si womate ŋu agblɔ ɖi o vɛ. Bu simulator ƒe emetsonu abe ɖoɖowɔwɔ ƒe anyigba si ŋu kakaɖedzi le ene — gazazã ƒe ɖoɖo na 10-15% ƒe taƒe bubu na nuwɔwɔ hehexɔxɔ ƒe duƒuƒu be woabu akɔnta le duƒuƒu ƒe tɔtrɔ ŋu.

Ðe simulator sia ɖea vi na asitɔtrɔ nyuie loo alo duƒuƒu blibo do ŋgɔ na hehexɔxɔ ko?

Eɖea vi ŋutɔ na wo ame evea siaa. Fine-tuning kple mɔnuwo abe LoRA alo QLoRA trɔa ŋkuɖodzinu ƒe nɔnɔme ŋutɔ elabena parameters ƒe akpa sue aɖe koe hiã gradients kple optimizer nɔnɔmewo. Simulator nyui aɖe na be nàte ŋu awɔ mɔnu siawo siwo zãa parameter-dɔwɔwɔ nyuie ƒe kpɔɖeŋu tẽ, si kpena ɖe ŋuwò nènya nenye be ɖɔɖɔɖodɔ nyui aɖe sɔ ɖe nuƒlela ɖeka ƒe GPU dzi alo bia GPU geɖe ƒe xɔtuɖoɖo.

Aleke esia do ƒome kple gazazãwo dzi kpɔkpɔ le asitsadɔwɔnuwo kple SaaS ƒe nudɔdɔwo katã me?

Gɔmeɖose vevitɔ — wɔ kpɔɖeŋu eye nàwɔ ɖoɖo ɖe nunɔamesiwo mama ŋu hafi atsɔ gazazã ade asi — wɔa dɔ le xexeame katã. Abe alesi ML ƒuƒoƒowo gblẽa akpe geɖe ɖe GPU siwo ŋu wowɔ ɖoɖo ɖo wògbɔ eme ŋu ene la, asitsahawo hã gblẽa akpe geɖe ɖe SaaS ƒe nudɔdɔ siwo tsia tre ɖe wo nɔewo ŋu kple dɔwɔnu ƒe kɔsɔkɔsɔ siwo me mama le ŋu. Ne èƒo wò dɔwɔwɔ ƒe ƒuƒoƒo nu ƒu ɖe mɔ̃ ɖeka me kple modular activation, alesi Mewayz te ɖe asitsatsa ƒe dɔwɔnuwo ŋu kple eƒe 207-module OS, ɖea viɖe siwo le dɔwɔwɔ nyuie me le wò GPU ŋkuɖodzinu ƒe mama ƒe lolome nyuitɔ me hafi hehexɔxɔ nadze egɔme la fiana.

| Dze wò dodokpɔ femaxee gɔme le app.mewayz.com eye nàtu dɔwɔwɔ ƒe ƒuƒoƒo si tututu wò ƒuƒoƒoa hiã.