Sisan, ne mago bɛ contexte minnu na, olu bɛɛ bɛ ne bolo. A to n ka bulɔgu sɛbɛn in sɛbɛn.

Sɛbɛnni danfara ni Python 3.14 ka ZSTD Module

ye

Python 3.14 bɛ compression.zstd modulu don gafemarayɔrɔ jɔnjɔn kɔnɔ, wa a bɛ sɛbɛnniw danfaracogo barikama kabakoma dɔ da wuli ni masin kalancogo misaliw tɛ. Ni i y’a suman cogo min na komɛsi bɛ se ka sɛbɛn fila sɔgɔ ɲɔgɔn na, i bɛ se k’u bɔɲɔgɔnko dɔn — fɛɛrɛ min bɛ wele ko Normalized Compression Distance (NCD) — wa sisan Zstandard b’a kɛ teliya la ka se ka kɛ fɛn dilanni baarakɛta ye.

sɛbɛnni danfara min sinsinnen bɛ sɛgɛsɛgɛli kan, o bɛ baara kɛ cogo di tiɲɛ na ?

Hakilina jɔnjɔn min bɛ danfara min sinsinnen bɛ degun kan, o ju bɔra kunnafoniko miiriya la. Ni compression algorithm i n’a fɔ Zstandard ye sɛbɛnni bloki dɔ sɔrɔ, a bɛ kɔnɔna daɲɛgafe dɔ jɔ min bɛ misaliw jira. Ni sɛbɛnni fla daɲɛw, daɲɛw fɔcogo ani u jɔcogo bɛ ɲɔgɔn na, n’i y’u Dòn ɲɔgɔn na, o bɛ na ni nɔ ye min ka bon dɔɔnin dɔrɔn ka Tɛmɛ sɛbɛnni belebeleba dɔrɔn de kan. N'u tɛ ɲɔgɔn na, hakɛ min bɛ ɲɔgɔn kan, o bɛ gɛrɛ hakɛ kelen-kelen fila bɛɛ lajɛlen na.

O jɛɲɔgɔnya in bɛ minɛ ni Normalized Compression Distance formula ye : NCD(x, y) = (C(xy) - min(C(x), C(y))) / max(C(x), C(y)), o la, C(x) ye sɛbɛnni x hakɛ degunlen ye, ani C(xy) ye sɛbɛn fila minnu sirilen don ɲɔgɔn na, olu hakɛ degunlen ye. NCD hakɛ min bɛ 0 kɛrɛfɛ, o kɔrɔ ye ko sɛbɛnw bɛ ɲɔgɔn bɔ kosɛbɛ, k’a sɔrɔ nafa min bɛ 1 kɛrɛfɛ, o kɔrɔ ye ko kunnafoni kɔnɔkow tɛ u ni ɲɔgɔn cɛ.

Min bɛ nin fɛɛrɛ in kɛ kabako ye, o ye ko a tɛ degeliw kunnafoniw de wajibiya, tokenization tɛ, embeddings tɛ, ani GPU tɛ. Komɛsi yɛrɛ bɛ Kɛ i n’a fɔ sɛbɛnni in jɔcogo misali kalanlen. Sɛgɛsɛgɛli minnu bɔra sɛbɛnw kɔnɔ i n’a fɔ "Low-Resource Text Classification: A Parameter-Free Classification Method with Compressors" (2023) y’a jira ko nCD min sinsinnen bɛ gzip kan, o ye BERT kɛlɛ jateminɛ dɔw kan, o kɛra sababu ye ka nafa kura don o fɛɛrɛ in na.

Mun na Python 3.14 ka Zstandard Module ye tulonkɛ-Yɛlɛma ye NCD kama ?

Sanni Python 3.14 ka se, Zstandard baara tun ka kan ka mɔgɔ sabanan python-zstandard pake sigi sen kan. compression.zstd modulu kura min donna sen kan PEP 784 fɛ, o bɛ ci ni CPython ye k’a ɲɛsin a yɛrɛ ma. O kɔrɔ ye ko zeru dependency overhead ani API garanti, sabatilen min bɛ dɛmɛn Meta ka kɛlɛ-kɔrɔbɔli libzstd fɛ. Kɛrɛnkɛrɛnnenya la, danfara-baaraw kama, Zstandard bɛ nafa caman Di gzip walima bzip2:

kan

• Teliya : Zstandard bɛ sɛgɛsɛgɛli kɛ teliya la siɲɛ 3-5 ka tɛmɛ gzip kan ni hakɛw ye minnu bɛ se ka suma ɲɔgɔn ma , o bɛ kɛ sababu ye ka batɛmu danfara kɛ sɛbɛn ba caman kan sekɔndi kɔnɔ sanni ka kɛ miniti damadɔ kɔnɔ
• Tunable compression levels : Nivo 1 fo 22 b’a to i bɛ teliya jago kɛ ni ratio ye , o b’a to i bɛ se ka NCD tiɲɛni jateminɛ ka kɛɲɛ ni tɛmɛsira wajibiyalenw ye
• Daɲɛgafe dɛmɛni : Zstandard daɲɛgafe minnu kalanna ka kɔn, olu bɛ se ka sɛbɛn misɛnninw deguncogo ɲɛ kosɛbɛ (4KB jukɔrɔ), o ye sɛbɛn hakɛ hakɛ ye tigitigi, NCD tiɲɛni nafa ka bon yɔrɔ min na kosɛbɛ
• Streaming API : Module bɛ dɔ fara compression kan , ka se ka danfara pipelines kɛ minnu bɛ sɛbɛnniw baara k’a sɔrɔ u ma corpora bɛɛ doni hakili la
• Bibliothèque standard sabatili : Version tɛ sɔsɔ , farati tɛ sɔrɔko cakɛda la — from compression import zstd bɛ baara kɛ Python 3.14+ sigicogo bɛɛ kan

ye

Hakilila jɔnjɔn : danfara min sinsinnen bɛ degun kan , o bɛ baara kɛ ka ɲɛ ni i mago bɛ basigi teliya la , min tɛ tali kɛ a la , min bɛ kan caman sɛbɛnniw ɲɛnabɔ i yɛrɛ la . Ikomi komɛsiw bɛ baara kɛ bayt raw kan sanni ka kɛ kan kɛrɛnkɛrɛnnen tokenw ye, u bɛ Sinuwakan, Arabukan walima kan ɲagaminen sɛbɛnw tila ka ɲɛ i n’a fɔ tubabukan — kanko misali si tɛ wajibiya.

ye

waleyali waleyali bɛ cogo di ?

NCD danfaralan fitinin min bɛ Python 3.14 kɔnɔ, o bɛ bɛn zana 30 jukɔrɔ. I bɛ kunnafoni sɛbɛn kelen-kelen bɛɛ kodɔn (kelen kulu kelen-kelen bɛɛ la), o kɔfɛ, sɛbɛn kura kelen-kelen bɛɛ kama, i bɛ NCD jate ka kɛɲɛ ni ɲɛfɔlisen kelen-kelen bɛɛ ye ani ka kulu di min ni yɔrɔjan ka dɔgɔn kosɛbɛ. Nin ye hakilina koloma ye :

Fɔlɔ, i ka modulu don ni ye ka bɔ compression import zstd la. Baarakɛcogo dɔ ɲɛfɔ min bɛ sɔn bayt sɛrɛ fila ma, ka kelen-kelen bɛɛ digidigi kelen-kelen, k’u ka ɲɔgɔndanw digidigi, ka NCD jate segin. O kɔfɛ, daɲɛgafe dɔ jɔ min bɛ kuluw kariti kɛ ka kɛɲɛ ni misali sɛbɛnw ye minnu bɛ mɔgɔ jira. Sɛbɛn kelen-kelen minnu bɛ na, aw bɛ segin-ka-bɔnye kɛ kuluw kan, ka NCD jatebɔ, ka a dɔgɔyalenba sugandi.

| Ni degun hakɛ kɔrɔtalen don ka se 10 ma, o bɛ tiɲɛni gɛlɛya ka se 68% ɲɔgɔn ma, o musaka ye ka dɔ bɔ baara kɛcogo la ka se sɛbɛn 2500 ɲɔgɔn ma segin kɔnɔ. Nin jatedenw tɛ bɛn transformateur fine-tuned ma, nka u bɛ basigi barikama di prototyping, data labeling triage, walima sigida minnu na ML dependencies sigili tɛ se ka kɛ.

NCD bɛ tali kɛ cogo di ni laadala ML danfara ye ?

Jaabi lakika ye ko NCD tɛ transformateur basigilen danfaralanw nɔnabila ye sɛnɛfɛnw dilanni siratigɛ la , minnu bɛ kɛ ni wari caman ye . Modeli minnu bɛ i n’a fɔ BERT walima GPT-based classifiers, olu bɛ 94%+ tiɲɛni sɔrɔ standard benchmarks kan. Nka, NCD ni Zstandard bɛ yɔrɔ kɛrɛnkɛrɛnnen dɔ minɛ. A ka fisa ni nɛnɛ daminɛko ye, misali minnu sɛbɛnnen bɛ i bolo, olu tɛ se 50 ma kalanso kelen kɔnɔ — ko min na hali modɛliw labɛnna ka ɲɛ, olu bɛ kɛlɛ kɛ. A bɛ degeliw waati zeru de wajibiya, a bɛ kan walima kodɔn suguya bɛɛ ɲɛnabɔ k’a sɔrɔ a ma fɛn caman sɛmɛntiya, wa a bɛɛ bɛ baara kɛ CPU kan ni hakilijagabɔ ye tuma bɛɛ.

| Nin pibiliki min bɛ kɛ ni dakun fila ye, o bɛ dɔ bɔ kosɛbɛ jateminɛ musakaw la k’a sɔrɔ a bɛ tiɲɛni bɛɛ lajɛlen mara. Plateforme minnu bɛ baarakɛlaw ka kunnafoniw baara ka kɛɲɛ ni sɛgɛsɛgɛli ye, i n’a fɔ Mewayz ka jago OS min bɛ kɛ ni modulu 207 ye, baarakɛla 138.000 ni kɔ bɛ baara kɛ ni min ye, olu bɛ nafa sɔrɔ danfara nɔgɔmanw na walasa ka cikanw bila sira kan, ka kunnafoniw taamasiyɛn, ani ka baarakɛlaw ka ko kɛlenw kɛ u yɛrɛ ta ye, k’a sɔrɔ fɛn girinmanw tɛ.

dan ni wale ɲumanw ye mun ye ?

dantigɛli min sinsinnen bɛ degun kan, o bɛ dan dɔntaw la i ka kan ka minnu jate. Sɛbɛn surunw (bayiti 100 jukɔrɔ) bɛ NCD jatebɔw Bɔ minnu tɛ Se ka Da u kan bawo kunnafoni bɛrɛ tɛ komɛsi la walasa ka misali kɔrɔw jɔ. Fɛɛrɛ in fana bɛ jateminɛ kɛ sɛbɛnw sugandili la — ciden minnu sugandira ka ɲɛ, olu bɛ tiɲɛni dɔgɔya kosɛbɛ. Wa ikomi NCD ye yɔrɔjan jateminɛ ye sanni ka kɛ misali ye min bɛ se ka kɛ, a tɛ dannaya jatew bɔ a yɛrɛ la.

Walisa ka nafa caman sɔrɔ nin fɛɛrɛ in na : baara kɛ ni sɛbɛnniw ye minnu bɛ se ka kɛ bayt 500 ye a dɔgɔyalenba la kulu kelen-kelen bɛɛ kɔnɔ, k’a lajɛ ni misali caman jɛli ye kalasi kelen kɔnɔ (sɛbɛn 2-3 minnu farala ɲɔgɔn kan, olu bɛ daɲɛgafe ɲumanw di minnu bɛ fɛnw sɛgɛsɛgɛ), ka sɛbɛnniw kɛyɔrɔw ni yɔrɔ finmanw kɛ cogo la min bɛ kɛ cogo la ka kɔn sɛgɛsɛgɛli ɲɛ, ani ka sɛgɛsɛgɛli kɛ Zstandard sɛgɛsɛgɛli hakɛ 3, 6 ani 10 kan walasa k’i ka teliya-tiɲɛni duman sɔrɔ. Walasa ka sɛbɛnni misɛnninw danfara, i ka Zstandard daɲɛgafe dɔ dege ka kɔn i ka domani corpus kan — nin sen kelen in bɛ se ka tiɲɛni ɲɛ ni kɛmɛsarada la 8-12 ye sɛbɛn surunw kan.

Ɲininkali minnu bɛ kɛ tuma caman na

yala danfara min sinsinnen bɛ degun kan, o bɛ baara kɛ dusukunnataw sɛgɛsɛgɛli kama wa ?

A bɛ se ka kɛ , nka ni lasɔminiw ye . Dusukunnataw sɛgɛsɛgɛli bɛ a ɲini ka fɔcogo danfara misɛnninw Kɔlɔsi sɛbɛnniw kɔnɔ minnu bɛ tali Kɛ ɲɔgɔn na u jɔcogo la. NCD bɛ baara kɛ ka ɲɛ barokunw danfara la yɔrɔ min na sɛbɛnw minnu bɛ kuluw kɔnɔ, olu bɛ baara kɛ ni daɲɛ danfaralenw ye. Dusukunnataw kama, a ka c’a la, tiɲɛni bɛ jigin 55-60% lamini na — a ka fisa ni k’a sɔrɔ a ma kɛ cogo si la, nka a tɛ labɛn ka ɲɛ a yɛrɛ ma. NCD fɛnw ni logistiki kɔsegin modɛli nɔgɔman faralen ɲɔgɔn kan, o bɛ sɔrɔw ɲɛ kosɛbɛ.

Yala n bɛ se ka baara kɛ ni compression.zstd modulu ye Python sɛrɛkiliw kɔnɔ ka kɔn 3.14 ɲɛ wa ?

Ayi. compression.zstd modulu ye ko kura ye Python 3.14 kɔnɔ. Fɔlɔw kama, python-zstandard pake sigi sen kan ka bɔ PyPI la, min bɛ compress() ni decompress() baarakɛcogo bɛnninw di. NCD ka hakilina bɛ to a cogo kelen na — import statement dɔrɔn de bɛ Changé. Ni i ye sɛgɛsɛgɛli kɛ ka se 3.14 ma, i bɛ se ka mɔgɔ sabanan ka dantigɛli bila pewu.

Zstandard NCD bɛ baara kɛ cogo di ni i y' a suma ni TF-IDF ye min ni kosin bɔlen don ?

Kalanso caman barokunw danfara kan ni kunnafonidilanw ye minnu bɛ bɛn ɲɔgɔn ma, TF-IDF faralen ɲɔgɔn kan cosine ɲɔgɔn fɛ, a ka c’a la, o bɛ se ka 75-82% tiɲɛ ni i y’a suma ni Zstandard NCD ta ye 62-68%. Nka, TF-IDF bɛ vektɔrikɛlan fitinin dɔ de wajibiya, daɲɛgafe ɲɛfɔlen, ani kan kɛrɛnkɛrɛnnenw ka daɲɛw jɔcogo lisɛli. Zstandard NCD tɛ nin baarakɛcogo fɔlɔ ninnu si wajibiya, a bɛ baara kɛ kanw kɔnɔ ka bɔ kɛsu kɔnɔ, wa a bɛ sɛbɛn kuraw tila waati basigilen kɔnɔ daɲɛw hakɛ mana kɛ min o min ye. Prototype teliya walima kan caman sigidaw kama, a ka c’a la, NCD ye sira teliya ye ka taa baarakɛyɔrɔ la.

I mana kɛ kɔnɔkow pipelines otomatikiw jɔli ye, ka kiliyanw ka cikanw bila sira kan, walima ka prototyping classification logic kɛ i ka jago nizɛrikan na, Python 3.14 ka Zstandard dɛmɛnan min bɛ a kɔnɔ, o bɛ NCD min sinsinnen bɛ compression kan, o bɛ se ka sɔrɔ ka tɛmɛn fɔlɔ kan. N’i bɛ fɛn bɛɛ lajɛlen yɔrɔ ɲini walasa k’i ka jago kɔnɔkow, i ka fɛn dilannenw, i ka kalanw, ani i ka kiliyanw ka jɛɲɔgɔnyaw ɲɛnabɔ, i ka jɔli daminɛ ni Mewayz ye bi ka nin fɛɛrɛ ninnu bila baara la i ka baara bɛɛ kɔnɔ.

tuma bɛɛ