Gutondekanya inyandiko hamwe na Python 3.14 Module ya ZSTD

Python 3.14 itangiza compression.zstd module mubitabo bisanzwe, kandi ifungura uburyo bukomeye butangaje muburyo bwo gutondekanya inyandiko nta moderi yo kwiga imashini. Mugupima uburyo compressor ishobora guhuza inyandiko ebyiri hamwe, urashobora kumenya ibyo bisa - tekinike yitwa Normalized Compression Distance (NCD) - none Zstandard ituma byihuta bihagije kubikorwa byakazi.

Nigute Kwandika Bishingiye ku Kwandika Byashyizwe mubikorwa?

Igitekerezo cyibanze cyihishe inyuma yo kwikuramo gushinga imizi mubitekerezo byamakuru. Iyo compression algorithm nka Zstandard ihuye nigice cyinyandiko, yubaka inkoranyamagambo yimbere yimiterere. Niba inyandiko ebyiri zisangiye amagambo asa, syntax, nuburyo, kubihuza hamwe bitanga ibisubizo binini cyane kuruta guhuza inyandiko nini yonyine. Niba bidafitanye isano, ingano ihuriweho yegeranye yegeranya igiteranyo cyubunini bwombi.

Iyi sano yafashwe na formulaire isanzwe yo guhuza intera: NCD (x, y) = (C (xy) - min (C (x), C (y))) / max (C (x), C (y)) , aho C (x) nubunini bwagabanijwe bwinyandiko x, kandi C (xy) nubunini bwanditse bwanditse. Agaciro NCD hafi 0 bivuze ko inyandiko zisa cyane, mugihe agaciro kegereye 1 bivuze ko basangiye hafi nta makuru yamakuru.

Igituma ubu buhanga butangaje nuko budasaba amakuru yamahugurwa, nta tokenisation, nta gushiramo, ndetse na GPU. Compressor ubwayo ikora nkicyitegererezo cyize cyimiterere yinyandiko. Ubushakashatsi bwasohotse mu mpapuro nka "Ibikoresho bito bito bito: Uburyo bwa Parameter-Itondekanya Uburyo hamwe na Compressors" (2023) bwerekanye ko NCD ishingiye kuri gzip yahanganye na BERT ku bipimo bimwe na bimwe, bituma abantu bashishikazwa n’uburyo.

Kuki Module ya Python 3.14 ari umukino-uhindura NCD?

Mbere ya Python 3.14, ukoresheje Zstandard bisaba kwishyiriraho igice cya gatatu python-zstandard . Module nshya compression.zstd module, yatangijwe binyuze kuri PEP 784, yoherejwe na CPython. Ibi bivuze kwishingikiriza kuri zeru hejuru hamwe na API yizewe, ihamye ishyigikiwe na Meta yintambara yageragejwe na libzstd. Kubikorwa byo gutondekanya byumwihariko, Zstandard itanga inyungu nyinshi kurenza gzip cyangwa bzip2:

• • > Kuberako compressor ikora kuri byite mbisi aho kuba ibimenyetso byihariye byururimi, bashyira mubyiciro byigishinwa, Icyarabu, cyangwa imvugo ivanze nkururimi rwicyongereza - nta cyitegererezo cyururimi gisabwa.

    Ishyirwa mubikorwa risa gute?

    Icyiciro gito cya NCD muri Python 3.14 gihuye numurongo 30. Uhindura buri nyandiko yerekana (imwe kuri buri cyiciro), hanyuma kuri buri nyandiko nshya, kubara NCD kuri buri cyerekezo hanyuma ugenera icyiciro hamwe nintera yo hasi. Dore ibitekerezo byibanze:

    Icyambere, kwinjiza module hamwe na kuva compression itumiza zstd . Sobanura imikorere yemera imirongo ibiri ya byte, ikomatanya buri muntu kugiti cye, ikomatanya guhuza, kandi igasubiza amanota ya NCD. Noneho wubake inkoranyamagambo ikarita yerekana ibirango byerekana icyitegererezo. Kuri buri nyandiko yinjira, subiramo ibyiciro, ubare NCD, hanyuma uhitemo byibuze.

    💡 DID YOU KNOW?

    Mewayz replaces 8+ business tools in one platform

    CRM · Invoicing · HR · Projects · Booking · eCommerce · POS · Analytics. Free forever plan available.

    Start Free →

    Mu bipimo birwanya amakuru yamakuru ya AG (amakuru yo mu byiciro bine byashyizwe mu byiciro), ubu buryo ukoresheje Zstandard kurwego rwo kwikuramo 3 bugera kuri 62-65% byukuri - nta ntambwe yo guhugura, nta gukuramo icyitegererezo, hamwe n’umuvuduko wo gutondekanya inyandiko zigera ku 8000 ku isegonda ku murongo umwe wa CPU. Kuzamura urwego rwo kwikuramo kugeza 10 bisunika neza kuri 68% mugiciro cyo kugabanya ibicuruzwa byinjira kugeza ku nyandiko 2500 kumasegonda. Iyi mibare ntabwo ihuye neza na transformateur nziza, ariko itanga urufatiro rukomeye rwa prototyping, data labels triage, cyangwa ibidukikije aho gushiraho ML biterwa bidashoboka.

    NCD igereranya ite na ML gakondo?

    Igisubizo cyinyangamugayo nuko NCD itari umusimbura wa transformateur-ishingiye kuri sisitemu yo gukora cyane. Icyitegererezo nka BERT cyangwa GPT ishingiye kubiciro bigera kuri 94% + byukuri kubipimo bisanzwe. Ariko, NCD hamwe na Zstandard ifite umwanya wihariye. Iratangaje mubihe bikonje-gutangira aho ufite ingero zitarenze 50 zanditseho ingero kumasomo - ibihe aho na moderi yatunganijwe neza irwana. Bisaba igihe cyamahugurwa ya zeru, ikora ururimi urwo arirwo rwose cyangwa kodegisi idahinduwe, kandi ikora rwose kuri CPU hamwe nububiko buhoraho.

    Kubucuruzi bucunga ibintu byinshi byinjira - itike yo gushyigikira, imbuga nkoranyambaga zivuga, gusubiramo ibicuruzwa - icyiciro cya Zstandard NCD gishobora gukora nka router ya mbere yerekana ibyiciro mugihe nyacyo mbere yuko moderi zihenze zitunganya ibisubizo. Uyu muyoboro wibyiciro bibiri ugabanya ibiciro byumwanzuro mugihe ugumye neza. Amahuriro atunganya abakoresha ibyakozwe mubipimo, nka Mewayz ya 207-module yubucuruzi ya Mewayz ikoreshwa na ba rwiyemezamirimo barenga 138.000, bungukirwa no gushyira mu byiciro byoroheje kugeza ubutumwa bwinzira, ibikubiyemo, no kumenyekanisha ubunararibonye bwabakoresha nta bikorwa remezo biremereye.

    Nibihe bigarukira hamwe nibikorwa byiza?

    Kwiyunvikana gushingiye kumurongo bifite aho bigarukira ugomba kubara. Inyandiko ngufi (munsi ya 100 bytes) zitanga amanota ya NCD atizewe kuko compressor idafite amakuru ahagije yo kubaka imiterere ifatika. Tekinike nayo yunvikana muguhitamo inyandiko zerekana - abahagarariye abatoranijwe nabi basuzugura neza ukuri. Kandi kubera ko NCD ari intera intera aho kuba icyitegererezo gishoboka, ntabwo isanzwe itanga amanota yicyizere.

    > Kubisobanuro bito-byanditse, banza uhugure inkoranyamagambo ya Zstandard kuri domaine yawe - iyi ntambwe imwe irashobora kunonosora neza amanota 8-12 ku ijana ku nyandiko ngufi.

    Ibibazo bikunze kubazwa

    Ese guhuza ibyiciro bishingiye kubikorwa byo gusesengura amarangamutima?

    Irashobora, ariko hamwe na caveats. Isesengura ryimyumvire risaba kumenya itandukaniro rito ryijwi mubyanditswe bisa. NCD ikora neza kubijyanye no gutondekanya ingingo aho inyandiko mubyiciro bitandukanye zikoresha amagambo atandukanye. Ku myumvire, ubunyangamugayo busanzwe bugera kuri 55-60% - biruta kubushake, ariko ntabwo byiteguye-byonyine. Guhuza ibiranga NCD hamwe na moderi yoroheje yo kwisubiramo yerekana neza ibisubizo neza.

    Nshobora gukoresha compression.zstd module muri verisiyo ya Python mbere ya 3.14?

    Oya. compression.zstd module ni shyashya muri Python 3.14. Kubisobanuro byambere, shyiramo pake ya python-zstandard kuva muri PyPI, itanga compress ihwanye () na decompress () imikorere. Logic ya NCD ikomeza kuba imwe - gusa ibyatumijwe mu mahanga birahinduka. Umaze kuzamura kuri 3.14, urashobora guta igice cya gatatu cyishingikirije rwose.

    Nigute Zstandard NCD ikora ugereranije na TF-IDF ihuriweho na cosine?

    Ku byiciro byinshi byiciro byashyizwe hamwe hamwe na datasets zuzuye, TF-IDF wongeyeho cosine isa mubisanzwe igera kuri 75-82% ugereranije na 62-68% ya Zstandard NCD. Ariko, TF-IDF isaba vectoriser ikwiye, amagambo asobanuwe, nurutonde rwamagambo yihariye. Zstandard NCD ntisaba na kimwe muribi gutunganya, ikora mu ndimi hanze yisanduku, kandi igashyira inyandiko nshya mugihe gihoraho utitaye ku bunini bw'amagambo. Kubyihuta prototyping cyangwa ibidukikije byindimi nyinshi, NCD ninzira yihuta kuri sisitemu ikora.

    Waba urimo wubaka imiyoboro yibikoresho byikora, kuyobora ubutumwa bwabakiriya, cyangwa prototyping classique logique kubucuruzi bwawe bwa digitale, inkunga ya Python 3.14 yubatswe muri Zstandard ituma NCD ishingiye kuri compression igerwaho kuruta mbere hose. Niba ushaka urubuga-rwose-umwe kugirango ucunge ibintu byubucuruzi, ibicuruzwa, amasomo, hamwe n’imikoranire yabakiriya, tangira kubaka hamwe na Mewayz uyumunsi hanyuma ushyire ubwo buhanga kugirango ukore mubikorwa byawe byose.