Ngoku ndinawo wonke umxholo endiwufunayo. Makhe ndibhale iposti yebhlog.

Ukuhlelwa kweSibhalo ngePython 3.14's ZSTD Module

I-Python 3.14 yazisa imodyuli compression.zstd kwilayibrari eqhelekileyo, kwaye ivula indlela enamandla ngokumangalisayo yokuhlelwa kombhalo ngaphandle kweemodeli zokufunda ngomatshini. Ngokulinganisa ukuba icompressor inokucudisa kangakanani iitekisi ezimbini kunye, unokumisela ukufana kwazo - ubuchule obubizwa ngokuba yiNormalized Compression Distance (NCD) - kwaye ngoku iZstandard iyenza ikhawuleze ngokwaneleyo kumthwalo wemveliso.

Lusebenza Njani uHlelo lweTeksi oluSekwe kuxinzelelo?

Ingcamango engundoqo emva kokuhlelwa okusekwe kucinezelo imiliselwe kwithiyori yolwazi. Xa i-algorithm yoxinzelelo efana ne-Zstandard idibana nebhloko yombhalo, yakha isichazi-magama sangaphakathi seepateni. Ukuba izicatshulwa ezibini zabelana ngesigama, isintaksi, kunye nolwakhiwo olufanayo, ukuzidibanisa kuvelisa isiphumo esikhulu kancinane kunokucinezela isicatshulwa esikhulu kuphela. Ukuba azizalani, ubungakanani obucinezelweyo obudityanisiweyo busondela kumdibaniso wazo zombini iisayizi zobukhulu obubodwa.

Obu budlelwane bubanjwe ngefomula ye-NCD (x, y) = (C(xy) - min(C(x), C(y))) / max(C(x), C(y)), apho uC(x) bubungakanani obucinezelweyo bombhalo x, kunye noC(xy) bubungakanani obucinezelweyo beetekisi ezimbini ezihlanganisiweyo. Ixabiso le-NCD elikufutshane no-0 lithetha ukuba izicatshulwa ziyafana kakhulu, ngelixa ixabiso elikufutshane no-1 lithetha ukuba babelane phantse akukho mxholo wolwazi.

Yintoni eyenza obu bubuchule buphawuleke kukuba ayifuni datha yoqeqesho, akukho phawu, akukho kuzinziswa, kwaye akukho GPU. Icompressor ngokwayo isebenza njengemodeli efundiweyo yesakhiwo sesicatshulwa. Uphando olupapashwe kumaphepha afana ne-"Low-Resource Text Classification: Indlela yokuHlela yeParameter-Free kunye neCompressors" (2023) ibonise ukuba i-NCD esekwe kwi-gzip ikhuphisana ne-BERT kwiibenchmarks ezithile, ivuselela umdla ohlaziyiweyo kwindlela.

Kutheni iPython 3.14's Zstandard Modyuli iTshintsho loMdlalo lwe-NCD?

Ngaphambi kwePython 3.14, usebenzisa i-Zstandard efunekayo ukufaka umntu wesithathu python-zstandard ipakethe. Imodyuli entsha compression.zstd, eyaziswa nge-PEP 784, ihambisa ngokuthe ngqo ngeCPython. Oku kuthetha ukuba akukho kuxhomekeka kwi-opharetha kunye ne-API eqinisekisiweyo, ezinzileyo exhaswa yi-libzstd evavanyiweyo yedabi le-Meta. Kwimisebenzi yokuhlela ngokukodwa, iZstandard ibonelela ngeenzuzo ezininzi ngaphezulu kwe-gzip okanye bzip2:

• Isantya: I-Zstandard icinezela 3-5x ngokukhawuleza kune-gzip ngokomlinganiselo othelekisekayo, yenza ukuhlelwa kweebhetshi kumawakawaka amaxwebhu asebenze ngemizuzwana kunemizuzu
• Amanqanaba oxinzelelo lwe-Tunable: Amanqanaba oku-1 ukuya kwe-22 akuvumela ukuba uthengise ngesantya somlinganiselo, okukuvumela ukuba ulungelelanise ukuchaneka kwe-NCD ngokuchasene neemfuno ze-throughput
• Inkxaso yesichazi-magama: Izichazi-magama eziqeqeshwe kwangaphambili ze-Zstandard zinokuphucula ngokumangalisayo ukucinezelwa kwemibhalo emincinci (phantsi kwe-4KB), elolona luhlu lobungakanani bamaxwebhu apho ukuchaneka kwe-NCD kubaluleke kakhulu
• I-API yokusasaza: Imodyuli ixhasa ucinezelo olongezelelekileyo, yenza imibhobho yolwahlulo isebenze eqhuba izicatshulwa ngaphandle kokulayisha yonke icorpora kwinkumbulo
• Uzinzo lwethala leencwadi elisemgangathweni: Akukho ngquzulwano kuguqulelo, akukho mngcipheko wonikezelo — ukusuka kuxinzelelo lokungenisa elizweni zstd isebenza kuyo yonke iPython 3.14+ ufakelo

Ingqondo engundoqo: Ukuhlelwa okusekwe kucinezelo kusebenza kakuhle xa ufuna isiseko esikhawulezayo, esingaxhomekekiyo esiphatha isicatshulwa seelwimi ezininzi ngokomthonyama. Ngenxa yokuba iicompressors zisebenza kwiibytes ekrwada kuneethokheni zolwimi oluthile, zihlela isiTshayina, isiArabhu, okanye amaxwebhu eelwimi ezixubeneyo kanye njengesiNgesi — akukho modeli yolwimi ifunekayo.

Sijongeka Njani uPhumezo oluSebenzayo?

Uluhlu oluncinci lwe-NCD kwiPython 3.14 lungena ngaphantsi kwemigca engama-30. Ufaka ikhowudi isicatshulwa ngasinye sereferensi (enye kwicandelo ngalinye), emva koko kuxwebhu olutsha ngalunye, ubale i-NCD ngokuchasene nereferensi nganye kwaye unike udidi ngowona mgama uphantsi. Nantsi eyona ngqiqo:

Kuqala, ngenisa imodyuli nge ukusuka kucinezelo lokungenisa zstd. Chaza umsebenzi owamkela iintambo ezimbini ze-byte, ucinezele ngamnye ngamnye, ucinezele ukudibanisa kwawo, kwaye ubuyisela amanqaku e-NCD. Emva koko yakha uluhlu lweelebhile zemephu yesichazi-magama kwisicatshulwa esimele isampula. Kuxwebhu ngalunye olungenayo, phinda ngaphezulu kweendidi, bala i-NCD, kwaye ukhethe ubuncinane.

Kwimilinganiselo echasene nedatha yedatha ye-AG (ukuhlelwa kweendaba zodidi olune), le ndlela yokusebenzisa i-Zstandard kwinqanaba loxinzelelo lwe-3 ifezekisa malunga ne-62-65% yokuchaneka - akukho nyathelo loqeqesho, akukho modeli yokukhuphela, kunye nesantya sokuhlelwa malunga namaxwebhu angama-8,000 ngesekhondi kwi-CPU enye engundoqo. Ukuphakamisa izinga lokunyanzeliswa kwe-10 tyhala ukuchaneka malunga ne-68% ngeendleko zokunciphisa umthamo ukuya kumaxwebhu angama-2,500 ngomzuzwana. La manani awahambelani neziguquli ezilungelelanisiweyo, kodwa zibonelela ngesiseko esomeleleyo seprototyping, ukuvavanywa kwedatha yokuleyibhela, okanye iimeko-bume apho ukufakela ukuxhomekeka kweML kungenakwenziwa.

Injani i-NCD xa ithelekiswa noHlelo lweML yesiNtu?

Impendulo enyanisekileyo yeyokuba i-NCD ayisiyonto ithatha indawo yabadidiyeli basekwe kwi-transformer kwiinkqubo zokuvelisa eziphezulu. Iimodeli ezifana ne-BERT okanye abahlalutyi basekwe kwi-GPT bafikelela kwi-94%+ yokuchaneka kwiibenchmarks eziqhelekileyo. Nangona kunjalo, i-NCD ene-Zstandard ithatha i-niche ekhethekileyo. Iyagqwesa kwiimeko zokuqala ezibandayo apho unemizekelo engaphantsi kwama-50 ephawulweyo kwiklasi nganye - imeko apho iimodeli ezilungelelanisiweyo ziyasokola. Ifuna iqanda lexesha loqeqesho, iphatha naluphi na ulwimi okanye ikhowudi ngaphandle kokuguqulwa, kwaye isebenza ngokupheleleyo kwi-CPU ngememori engaguqukiyo.

Kumashishini alawula umthamo omkhulu womxholo ongenayo - amathikithi axhasayo, ukukhankanywa kweendaba zoluntu, ukuphononongwa kwemveliso - i-Zstandard NCD classifier inokusebenza njenge-router yokuqala yokupasa eyahlula amaxwebhu ngexesha langempela ngaphambi kokuba iimodeli ezibiza kakhulu ziphucule iziphumo. Lo mbhobho wamanqanaba amabini unciphisa iindleko zokuthelekelela kakhulu ngelixa ugcina ukuchaneka ngokubanzi. Amaqonga alungisa umxholo oveliswe ngumsebenzisi ngokomlinganiselo, njenge-OS ye-Mewayz's 207-modyuli yeshishini esetyenziswa ngoosomashishini abangaphezu kwe-138,000, baxhamla kuhlelo olukhaphukhaphu ukuya kwimiyalezo yendlela, umxholo wethegi, kwaye wenze amava omsebenzisi alungele wena ngaphandle kweziseko zophuhliso ezinzima.

Iyintoni imida kunye nezenzo ezilungileyo?

Uhlelo olusekwe kucinezelo lunolwazi lwemida omele uphendule ngayo. Imibhalo emifutshane (ngaphantsi kwe-100 bytes) ivelisa amanqaku angathembekanga e-NCD kuba icompressor ayinayo idatha eyaneleyo yokwakha iipatheni ezinentsingiselo. Ubuchule bukwanovakalelo kukhetho lweetekisi zereferensi - abameli abakhethwe kakubi bathoba ukuchaneka kabukhali. Kwaye ngenxa yokuba i-NCD iyimetric yomgama kunemodeli enokwenzeka, ayivelisi amanqaku okuzithemba ngokwemvelo.

Ukufumana okuninzi kule ndlela yokwenza: sebenzisa iitekisi zesalathiso ezinobuncinci be-500 bytes kwicandelo ngalinye, ulinge ngokudibanisa imizekelo emininzi kwiklasi nganye (ama-2-3 amaxwebhu amele adityaniswe kunye avelisa izichazi-magama ezingcono zoxinzelelo), lungisa i-text casing kunye ne-whitespace phambi koxinzelelo, kunye nebenchmark kuwo wonke amanqanaba oxinzelelo lwe-Zstandard 3, 6, kunye ne-10 ukufumana ukuchaneka kwesantya esimnandi. Kuhlelo lombhalo omncinci, qeqeshelwa kwangaphambili isichazi-magama se-Zstandard kwi-domain corpus yakho — eli nyathelo linye linokuphucula ukuchaneka ngepesenti ezi-8-12 zamanqaku kumaxwebhu amafutshane.

Imibuzo Ebuzwa Rhoqo

Ngaba ulwahlulo olusekwe kuxinzelelo lusebenza kuhlalutyo lweemvakalelo?

Inako, kodwa kunye nezilumkiso. Uhlalutyo lweemvakalelo lufuna ukufumanisa iiyantlukwano zethoni ezifihlakeleyo phakathi kweetekisi ezifanayo. I-NCD isebenza bhetele kuhlelo lwesihloko apho amaxwebhu akwiindidi ezahlukeneyo asebenzisa isigama esahlukileyo. Ngoluvo, ukuchaneka ngokuqhelekileyo kuhlala malunga ne-55-60% - ngcono kunokungakhethi, kodwa ingekho imveliso-elungele ngokwayo. Ukudibanisa iifitsha ze-NCD kunye nemodeli ekhaphukhaphu yokubuyisela uhlengahlengiso kuphucula iziphumo kakhulu.

Ndingasebenzisa imodyuli ye-compression.zstd kwiinguqulelo zePython ngaphambi kwe-3.14?

Hayi. Imodyuli compression.zstd intsha kwiPython 3.14. Kwiinguqulelo zangaphambili, faka python-zstandard iphakheji esuka kwiPyPI, ebonelela ngokulinganayo compress() kunye decompress() imisebenzi. Ingqiqo ye-NCD ihlala ifana - kuphela ingxelo yokungenisa eguqukayo. Nje ukuba uphucule ukuya ku-3.14, unokuyeka ukuxhomekeka komntu wesithathu ngokupheleleyo.

Isebenza njani iZstandard NCD xa ithelekiswa neTF-IDF ngokufana kwecosine?

Kuhlelo lwesihloko esineeklasi ezininzi ezineesethi zedatha ezilungelelanisiweyo, i-TF-IDF kunye nokufana kwecosine kudla ngokufikelela kuma-75-82% ukuchaneka xa kuthelekiswa ne-Zstandard NCD's 62-68%. Nangona kunjalo, i-TF-IDF ifuna i-vectoriser efakelweyo, isigama esichaziweyo, kunye noluhlu lwesitokhwe solwimi oluthile. I-Zstandard NCD ayifuni nanye kolu lungiso, isebenza kuzo zonke iilwimi ngaphandle kwebhokisi, kwaye ihlela amaxwebhu amatsha ngexesha elingaguqukiyo nokuba singakanani na isigama. Kwimeko ekhawulezayo yokwenziwa kweprototyping okanye iilwimi ezininzi, i-NCD ihlala iyindlela ekhawulezayo yesixokelelwano esisebenzayo.

Enoba wakha imibhobho yomxholo ozenzekelayo, uthumela imiyalezo yabathengi, okanye ulungelelanisa ulwahlulo lwe-prototyping kwishishini lakho ledijithali, inkxaso ye-Python 3.14 eyakhelwe-ngaphakathi ye-Zstandard yenza i-NCD esekelwe kuxinzelelo ifikeleleke ngakumbi kunangaphambili. Ukuba ujonge iqonga lee-in-one lokulawula umxholo weshishini lakho, iimveliso, izifundo, kunye nentsebenziswano yabathengi, qalisa ukwakha ngeMewayz namhlanje kwaye ubeke obu buchule busebenze kuwo wonke umsebenzi wakho.