Manje senginawo wonke umongo engiwudingayo. Ake ngibhale okuthunyelwe kwebhulogi.

Ukuhlelwa Kombhalo nge-Python 3.14's ZSTD Module

I-

Python 3.14 yethula imojuli ye-compression.zstd kulabhulali evamile, futhi ivula indlela enamandla ngokumangalisayo yokuhlukanisa umbhalo ngaphandle kwamamodeli okufunda omshini. Ngokukala ukuthi i-compressor ingakwazi ukuminyanisa kahle kangakanani imibhalo emibili ndawonye, ungakwazi ukunquma ukufana kwayo - indlela ebizwa ngokuthi i-Normalized Compression Distance (NCD) - futhi manje i-Zstandard iyenza isheshe ngokwanele ukulayisha umsebenzi wokukhiqiza.

Kusebenza Kanjani Ukuhlelwa Kombhalo Okusekelwe Ngokuminyanisa Ngempela?

Umqondo oyinhloko wokuhlukanisa okusekelwe ekucindezelekeni usekelwe kuthiyori yolwazi. Uma i-algorithm yokucindezela efana ne-Zstandard ihlangabezana nebhulokhi yombhalo, yakha isichazamazwi sangaphakathi samaphethini. Uma imibhalo emibili yabelana ngesilulumagama, i-syntax, kanye nesakhiwo esifanayo, ukuyihlanganisa ndawonye kukhiqiza umphumela omkhulu kancane kunokucindezela umbhalo omkhulu wodwa. Uma zingahlobene, usayizi ocindezelwe ohlanganisiwe usondela kwisamba sabo bobabili osayizi abangabodwana.

Lobu budlelwano buthwetshulwa ngefomula Yebanga Lokucindezela Okujwayelekile: NCD(x, y) = (C(xy) - min(C(x), C(y))) / max(C(x), C(y)), lapho u-C(x) engusayizi ocindezelwe wombhalo x, futhi u-C(xy) uwusayizi ocindezelwe wemibhalo emibili ehlanganisiwe. Inani le-NCD eliseduze no-0 lisho ukuthi imibhalo ifana kakhulu, kuyilapho inani eliseduze no-1 lisho ukuthi awabelani nhlobo ngokuqukethwe kolwazi.

Okwenza le nqubo iphawuleke ukuthi ayidingi idatha yokuqeqeshwa, akukho ukwenziwa kwamathokheni, akukho okushumekiwe, futhi akukho GPU. Icompressor ngokwayo isebenza njengemodeli efundiwe yesakhiwo sombhalo. Ucwaningo olushicilelwe emaphepheni afana ne-"Low-Resource Text Classification: A Parameter-Free Classification Method with Compressors" (2023) lubonise ukuthi i-Gzip-based NCD iqhudelana ne-BERT kumabhentshimakhi athile, okuvusa intshisekelo evuselelwe endleleni.

Kungani i-Python 3.14's Zstandard Module I-Game-Changer ye-NCD?

Ngaphambi kwe-Python 3.14, kusetshenziswa i-Zstandard idinga ukufaka iphakheji yenkampani yangaphandle python-zstandard. Imojula entsha ye-compression.zstd, eyethulwe nge-PEP 784, ihamba ngokuqondile nge-CPython. Lokhu kusho ukuthi i-zero dependency overhead kanye ne-API eqinisekisiwe, ezinzile esekelwa i-libzstd ehlolwe impi ye-Meta. Ngemisebenzi yokuhlukanisa ngokuqondile, i-Zstandard inikeza izinzuzo ezimbalwa ngaphezu kwe-gzip noma bzip2:

• Isivinini: I-Zstandard iminyanisa ngo-3-5x ngokushesha kune-gzip ngezilinganiso eziqhathanisekayo, okwenza ukuhlukaniswa kwenqwaba ngaphezu kwezinkulungwane zamadokhumenti kusebenze ngamasekhondi kunemizuzu
• Amazinga okucindezela avulelekayo: Amaleveli 1 kuya ku-22 akuvumela ukuthi uhwebe ngesivinini ukuze uthole isilinganiso, okukuvumela ukuthi ulinganise ukunemba kwe-NCD ngokumelene nezidingo zokukhipha amandla
• Usekelo lwesichazamazwi: Izichazamazwi ze-Zstandard eziqeqeshwe kusengaphambili zingathuthukisa ngokumangazayo ukuminyaniswa kwemibhalo emincane (ngaphansi kuka-4KB), okuyibanga ncamashi losayizi wedokhumenti lapho ukunemba kwe-NCD kubaluleke kakhulu khona
• I-API Yokusakaza: Imojula isekela ukucindezela okukhulayo, ukunika amandla amapayipi okuhlukanisa acubungula imibhalo ngaphandle kokulayisha yonke inhlangano kumemori
• Ukuzinza kwelabhulali okujwayelekile: Azikho izingxabano zenguqulo, azikho izingcuphe ze-supply chain — kusuka kokuminyanisa ukungenisa zstd isebenza kukho konke ukufakwa kwe-Python 3.14+

Imininingwane eyinhloko: Ukuhlelwa okusekelwe ekucindezelweni kusebenza kahle kakhulu uma udinga isisekelo esisheshayo, esinganciki esiphatha umbhalo wezilimi eziningi ngokomdabu. Ngenxa yokuthi ama-compressor asebenza kuma-byte aluhlaza esikhundleni samathokheni aqondene nolimi oluthile, ahlukanisa amadokhumenti wesiShayina, isi-Arabhu, noma izilimi ezixubile ngendlela efanele njengesiNgisi — ayikho imodeli yolimi edingekayo.

Kubukeka Kanjani Ukuqaliswa Okungokoqobo?

Isigaba esincane se-NCD ku-Python 3.14 silingana ngaphansi kwemigqa engu-30. Ubhala umbhalo ngamunye oyisithenjwa (owodwa esigabeni ngasinye), bese kudokhumenti entsha ngayinye, ubale i-NCD kuzo zonke izithenjwa futhi unikeze isigaba ngebanga eliphansi kakhulu. Nansi ingqondo ewumongo:

Okokuqala, ngenisa imojuli nge-kusuka kokucindezelwa ukungenisa zstd. Chaza umsebenzi owamukela amayunithi ezinhlamvu amabili we-byte, ucindezela ngayinye ngayinye, ucindezela ukuhlanganisa kwawo, futhi ubuyisele isikolo se-NCD. Bese wakha amalebula esigaba sokumepha isichazamazwi ukuze abe amasampula amele imibhalo. Ngombhalo ngamunye ongenayo, phindaphinda izigaba, bala i-NCD, bese ukhetha okungenani.

Kumabhentshimakhi aphikisana nedathasethi yezindaba ze-AG (ukuhlukaniswa kwezindaba kwezigaba ezine), le ndlela esebenzisa i-Zstandard ezingeni lokuminyanisa 3 ifinyelela cishe ukunemba okungu-62-65% — asikho isinyathelo sokuqeqesha, akukho modeli yokulanda, nesivinini sokuhlukanisa samadokhumenti acishe abe ngu-8,000 ngesekhondi kungqikithi eyodwa ye-CPU. Ukwenyusa izinga lokucindezelwa libe ngu-10 kuphusha ukunemba kuye cishe ku-68% ngezindleko zokunciphisa ukuphuma kube imibhalo engaba ngu-2,500 ngomzuzwana. Lezi zinombolo azifani neziguquli ezicushwe kahle, kodwa zinikeza isisekelo esiqinile se-prototyping, ukunqunywa kwamalebula kwedatha, noma izindawo lapho ukufakwa kwe-ML kungenakusebenza khona.

Iqhathaniswa Kanjani I-NCD Nokuhlelwa Kwe-ML Yendabuko?

Impendulo ethembekile ukuthi i-NCD ayithatheli indawo yezigaba ezisekelwe ku-transformer ezinhlelweni zokukhiqiza ezisezingeni eliphezulu. Amamodeli afana ne-BERT noma izihlukanisi ezisekelwe ku-GPT afinyelela ukunemba okungu-94%+ kumabhentshimakhi ajwayelekile. Kodwa-ke, i-NCD ene-Zstandard ithatha i-niche eyingqayizivele. Ihamba kahle kakhulu ezimweni ezibandayo lapho unezibonelo ezinelebula ezingaphansi kuka-50 ngekilasi ngalinye - isimo lapho ngisho namamodeli acushwe kahle adonsa kanzima. Idinga iqanda lesikhathi sokuqeqeshwa, isingatha noma yiluphi ulimi noma umbhalo wekhodi ngaphandle kokulungiswa, futhi isebenza ngokuphelele ku-CPU enenkumbulo engaguquki.

Kumabhizinisi aphethe umthamo omkhulu wokuqukethwe okungenayo — amathikithi okusekela, ukukhuluma ngenkundla yezokuxhumana, ukubuyekezwa kwemikhiqizo — isigaba se-Zstandard NCD singasebenza njengomzila ohamba phambili ohlukanisa amadokhumenti ngesikhathi sangempela ngaphambi kokuba amamodeli abizayo alungise imiphumela. Lo mzila wepayipi wezigaba ezimbili wehlisa izindleko zokucatshangelwa ngokuphawulekayo kuyilapho ugcina ukunemba okuphelele. Amapulatifomu acubungula okuqukethwe okukhiqizwe umsebenzisi ngezinga, okufana ne-OS yebhizinisi lamamojula angu-207 ka-Mewayz esetshenziswa osomabhizinisi abangaphezu kuka-138,000, bayazuza kusukela ekuhlukaniseni okunesisindo esincane ukuya emilayezweni yomzila, okuqukethwe komaka, nokwenza kube ngokwakho okuhlangenwe nakho komsebenzisi ngaphandle kwengqalasizinda esindayo.

Iyini Imikhawulo Nemikhuba Engcono Kakhulu?

Ukuhlelwa okusekelwe ekucindezelweni kunemikhawulo eyaziwayo okufanele uyicabangele. Imibhalo emifushane (ngaphansi kwamabhayithi angu-100) ikhiqiza amaphuzu e-NCD angathembekile ngenxa yokuthi umminyanisi awunayo idatha eyanele yokwakha amaphethini anengqondo. Isu liphinde libe nozwelo ekukhetheni imibhalo eyizethenjwa - abameleli abakhethwe kabi behlisa isithunzi sokunemba kakhulu. Futhi ngenxa yokuthi i-NCD iyimethrikhi yebanga kunemodeli engenzeka, ayikhiqizi amaphuzu okuzethemba ngokwemvelo.

Ukuze uthole okuningi kule ndlela yokwenza: sebenzisa imibhalo eyireferensi okungenani engamabhayithi angu-500 esigabeni ngasinye, zama ngokuhlanganisa izibonelo eziningi ngekilasi ngalinye (amadokhumenti amele ama-2-3 ahlanganiswe ndawonye akhiqiza izichazamazwi ezicindezelayo ezingcono kakhulu), yenza ukwakheka kombhalo kube okujwayelekile kanye nesikhala esimhlophe ngaphambi kokuminyanisa, kanye nebhentshimakhi kuwo wonke amaleveli okuminyanisa we-Zstandard 3, 6, no-10 ukuze uthole ukunemba okumnandi kwesivinini sakho. Ukuze uthole ukuhlelwa kombhalo omncane, qeqeshela kusengaphambili isichazamazwi se-Zstandard esizindeni sakho ikhophasi — lesi sinyathelo esisodwa singathuthukisa ukunemba ngamaphesenti angu-8-12 kumadokhumenti amafushane.

Imibuzo Evame Ukubuzwa

Ingabe ukuhlukaniswa okusekelwe ekucindezelweni kusebenza ekuhlaziyweni kwemizwelo?

Ingakwazi, kodwa ngezixwayiso. Ukuhlaziya imizwa kudinga ukuthola umehluko ocashile wephimbo phakathi kwamathekisthi afanayo ngesakhiwo. I-NCD isebenza kangcono ekuhlukaniseni isihloko lapho imibhalo emikhakheni ehlukene isebenzisa amagama ahlukene. Ngomzwelo, ukunemba ngokuvamile kuhlala cishe ku-55-60% - okungcono kunokungahleliwe, kodwa hhayi ukukhiqiza-okulungele ngokwakho. Ukuhlanganisa izici ze-NCD nemodeli yokuhlehliswa kwezinto engasindi kuthuthukisa imiphumela kakhulu.

Ngingakwazi ukusebenzisa imojuli ye-compression.zstd ezinguqulweni zePython ngaphambi kuka-3.14?

Cha. Imojuli ye-compression.zstd yintsha ku-Python 3.14. Ngezinguqulo zangaphambilini, faka iphakheji ye-python-zstandard esuka ku-PyPI, ehlinzeka ngokulinganayo compress() kanye decompress() imisebenzi efanayo. Umqondo we-NCD uhlala ufana - yisitatimende sokungenisa kuphela esishintshayo. Uma usuthuthukele ku-3.14, ungakwazi ukuyeka ngokuphelele ukuncika kwenkampani yangaphandle.

Isebenza kanjani i-Zstandard NCD uma iqhathaniswa ne-TF-IDF ngokufana kwe-cosine?

Ekuhlukanisweni kwezihloko zezigaba eziningi ezinamadathasethi abhalansi, ukufana kwe-TF-IDF kanye ne-cosine ngokuvamile kuzuza ukunemba okungu-75-82% uma kuqhathaniswa ne-Zstandard NCD's 62-68%. Kodwa-ke, i-TF-IDF idinga i-vectoriser efakiwe, isilulumagama esichaziwe, nohlu lwezinqamuli zolimi oluthize. I-Zstandard NCD ayikudingi kulokhu kucutshungulwa ngaphambilini, isebenza kuzo zonke izilimi ngaphandle kwebhokisi, futhi ihlukanisa amadokhumenti amasha ngesikhathi esingaguquki kungakhathaliseki usayizi wesilulumagama. Ezindaweni ze-prototyping ezisheshayo noma ezinezilimi eziningi, i-NCD ivamise ukuba yindlela esheshayo eya ohlelweni lokusebenza.

Kungakhathaliseki ukuthi wakha amapayipi okuqukethwe okuzenzakalelayo, uqondisa imilayezo yekhasimende, noma ukuhlelwa kwe-prototyping yebhizinisi lakho ledijithali, usekelo olwakhelwe ngaphakathi lwe-Python 3.14 lwe-Zstandard lwenza i-NCD esekelwe ekucindezeleni ifinyeleleke kakhulu kunangaphambili. Uma ufuna inkundla ehlangene yonke into yokuphatha okuqukethwe kwebhizinisi lakho, imikhiqizo, izifundo, nokusebenzisana kwamakhasimende, qala ukwakha nge-Mewayz namuhla futhi usebenzise lezi zindlela ukuze zisebenze kuwo wonke umsebenzi wakho.