Riba:
- Ƙididdigar ƙima mai ƙima da ƙima, haɗaɗɗiya da ma'amalar ɓarna
- Zaɓi dabarun chunking da suka dace don nau'ikan takardu daban-daban (PDF, tebur, lamba, log ɗin taɗi)
- Ƙarfafa ingancin maidowa da tacewa ta ƙara metadata zuwa kowane yanki
Wannan shine mafi girman rashin kulawa amma mafi yanke hukunci a cikin RAG: yadda kuke rushe takaddar. Ana kiran wannan chunking. Ko da kun ba da takarda iri ɗaya ga samfurin iri ɗaya, saboda mummunan chunking maidowa ya dawo da abin da bai dace ba kuma samfurin ba zai taɓa samar da babbar amsa ba. A cikin wannan rukunin, mun rufe dabarun rarrabuwa, yadda ake daidaita su bisa ga nau'in takarda, da yadda ake ƙara metadata mai ma'ana ga kowane guntu.
Me yasa Muke Shure?
Akwai dalilai guda uku. Na farko, haɗa nau'ikan suna canza rubutu har zuwa wani tsayin daka zuwa ma'ana mai ma'ana; Idan babi mai shafi 40 gabaɗaya ya cukuɗe a cikin vector guda ɗaya, ma'anar ta zama "batsewa." Na biyu, muna so mu ba da samfurin kawai ɓangaren da ake buƙata a matsayin mahallin; mika takardar gaba daya yana da tsada da jan hankali. Na uku, don maidowa ya zama daidai, sashin bincike dole ne ya zama ƙarami kuma a mai da hankali.
Don haka chunk shine mafi ƙarancin naúrar maidowa. Bai kamata ya zama babba ko ƙarami ba - dai dai.
Girman Chunk da Ma'aunin Matsala
Akwai manyan saituna guda biyu: girman chunk (alamu/kalmomi nawa ne za su kasance a cikin gungu) da kuma zoba (bangaren da chunks makwabta ke raba).
Ƙananan gungu (misali alamu 100): mai da hankali amma an cire haɗin daga mahallin. Ya ce "har tsawon kwanaki 14", amma abin da ya rage saura kwanaki 14 a cikin jumlar da ta gabata. Manyan chunks (misali alamu 2000): yana adana mahallin amma an gauraye zaren da yawa a ciki; haɗawa yana samun ɓarna kuma batutuwan da ba su da mahimmanci sun taru.
Haɗuwa yana magance matsalar iyaka. Idan jumla ta fadi daidai kan iyakar sassa biyu, sai a raba ta gida biyu ba tare da tafe ba kuma ta rasa ma’anarta. Haɓaka alamun 50-100 yana tabbatar da cewa bayanin da ke faɗowa cikin iyaka ya ci gaba da kasancewa a cikin aƙalla sashi ɗaya.
Girman tsinke
Amfani
Hasara
dace abun ciki
Ƙananan (alamu 100-250)
Babban hankali, mai da hankali
Magana na iya karya
FAQ, gajerun labarai, ma'anoni
Matsakaici (alamu 300-600)
Ma'auni; mafi yawan al'amuran
-
Tsari, rubutun manufofin
Babban (alamu 800-1500)
Mutuncin yanayi
m sakawa
Labari, dogon bayani
Tukwici: Idan ba ku san inda za ku fara ba, fara da ɓangarorin alamar 400-500 da 50-80 token zoba; sannan ku auna ku daidaita da bayanan ku. Girman "daidai" ba na duniya ba ne, ya dogara da mahallin.
Dabarun Cunking
Kafaffen-girman: Yana gyara rubutu kowane N tokens. Yana da sauƙi kuma mai sauri, amma yana iya katse tsakiyar jumla.
Mai Rarraba (mai maimaitawa/masu raba): Rarraba bisa ga sakin layi sannan kuma iyakokin jumla; Ya fi kiyaye mutuncin ma'ana. Yawancin tsarin samarwa suna farawa da wannan.
Jumla na Semantic: Yana kallon abin da aka haɗa jimlolin kuma ya raba su inda batun canjin ya faru. Hanya ce mafi inganci amma mafi tsada; Tare da babban kundin, farashin ma'amala yana ƙaruwa.
Tsari-sane: Yana amfani da tsarin daftarin aiki kamar kanun labarai, sassan, teburi. Misali, raba daftarin aiki ta Markdown ta kanun labarai yana tabbatar da cewa kowane bangare yana dauke da kan sa.
Daidaitawa ta Nau'in Takardu
Ba kowace takarda ba iri ɗaya ce. Dabarun sun bambanta da nau'in:
- PDF/Rubutun doka: tushen alamar shafi, matsakaicin girman. Share shafi sama/ƙasa mai maimaitawa (mai kai/ƙafa).
- Tables: Kada ku ɗauki layi daga mahallin; kiyaye kowane jere tare da bayanin kai ("Abu: X, Farashin: Y, Stock: Z"). Mayar da ɗanyen teburi zuwa rubutu a sarari yana da mahimmanci sau da yawa.
- Lambar: Raba ta iyakokin aiki / aji; Kada ku yanke wani aiki daga hanya.
- Taɗi/ Rikodin tikiti: Raba ta saƙo ko zagayowar zance; Ci gaba da sanin wanda ya ce me.
# gungu-gungu na tushen bracket (conceptual) chunks = bol (rubutu, manufa_size = 450, # token overlap=70, # token brackets = ["\n\n", "\n", "."," "] # sakin layi na farko, kalma ta ƙarshe)
Ƙara metadata zuwa kowace Waƙa
Tsaki ba wai kawai “raba” ba ne; shine a wadata kowane yanki. Kowane alamar da kuka haɗa wa waƙar yana da ƙimar nauyinsa a cikin zinare don tacewa da kuma ambaton tushen gaba.
# Ingantattun chunk (conceptual){"rubutu": "Izinin biya na shekara-shekara shine kwanaki 14 tare da sabis na shekaru 1-5...", "metadata": {"source": "ik_el_kitabi_v7.pdf", "section": "5.2 Annual Leave", "shafi": 23, "kwanan wata": "6025K": "6025K" "na ciki"}}
Wata dabara mai ƙarfi ita ce ƙara taken mahallin: rubuta taken babin da yake cikinsa a farkon kowane yanki. Don haka, ko da wani yanki da aka raba kamar "Domin kwanaki 14" duka biyun sun fi dacewa kuma suna da ma'ana kamar "Barka da shekara - kwanaki 14."
Rauni mai ƙarfi / Ƙarfin Ƙarfi
Rauni (makafin wuya, babu metadata):
Yanke rubutu kowane haruffa 1000. Ajiye rubutun kawai.# Sakamako: Tables sun rabu a tsakiya, "kwanaki 14" ya rage ba tare da mahallin ba, # ba a san ko wane takarda ya fito ba, ba za a iya yin tacewa ba.
Mai ƙarfi (tsari-sane + kai + metadata):
Raba daftarin aiki ta kanun labarai; ƙara taken sashe zuwa kowane bangare;haɗa tushen, shafi, kwanan wata da metadata keɓantawa; canza teburi zuwa rubutu na fili tare da taken su. # Sakamako: mai da hankali, mahallin mahallin, mai tacewa, mai tushe.
Mini Cases guda uku
Case 1 - Bala'i na zane. Ƙungiyar kuɗi ta raba jerin farashi mai shafuka 200 tare da yanke makanta mai wuya; An raba layuka na tebur ba da gangan ba. "Mene ne farashin samfur X?" Samfurin ya karanta layin da ba daidai ba kuma ya ba da farashi mara kyau (9 cikin 12 ba daidai ba ne). Lokacin da na canza layuka na tebur zuwa rubutu na fili a cikin tsarin "Samfur: … | Farashin:… | Raka'a: ..." kuskuren ya ragu zuwa 0 cikin 12.
Case 2 - Matsanancin babban gunki. A cikin wiki, kowane shafi an yi shi da gunki ɗaya (wasu suna cewa alamu 3,000). Abin da aka saka yana da duhu saboda akwai "leave", "overtime" da "payroll" a shafi ɗaya; Sashen sa'o'in aiki kuma ya shigo cikin wasa game da tambayar hutu. Lokacin da aka raba shafukan zuwa matsakaicin girman ta take, tuno@5 ya ƙaru daga 64% zuwa 91%.
Hali na 3 - Yanke jumla ba tare da zoba ba. 250 ƙayyadaddun alamar yanke don ƙungiyar lauyoyi, babu zoba. Ma'anar mahimmanci ta faɗi daidai a kan iyakar sassa biyu kuma ta rabu biyu; Ko daya ko daya bai ƙunshi cikakkiyar amsar ba. Lokacin da aka ƙara zoba 60, ma'anar guda ɗaya ta kasance a cikin yanki ɗaya kuma an mayar da amsar daidai.
Kuskuren gama gari
- Yanke makafi: Rarraba jimloli da teburi a tsakiya; ma'ana ta ɓace.
- Barin zoba a sifili: Bayanin da ya faɗo kan iyaka yana raba kuma ya ɓace.
- Ba ƙara metadata ba: Tacewa da nunin tushe sun zama ba zai yiwu ba.
- Barin tebur danye: Samfurin ba zai iya warware tsarin tebur ba; Maida layi zuwa rubutu na fili.
- Ƙaddamar da dabara ɗaya: PDF, code da tebur ba a raba su ta hanya ɗaya; Daidaita zuwa nau'i.
Tsanaki: Kar a saita Chunking sau ɗaya kuma manta da shi. Sake auna ingancin maidowa yayin da sabbin nau'ikan takardu suka zo (tikiti daga sabon tsarin, PDFs da aka duba). Bayanan shigarwa mara kyau yana nufin mummunan amsa ("datti a ciki, datti").
A takaice
- Chunk shine mafi ƙarancin naúrar maidowa; Ba ma babba ko ƙarami ba - ya kamata a daidaita daidai da abun ciki.
- Girman chunk yana nuna ma'auni-ma'auni; Zoba yana sarrafa asarar iyaka.
- Tsage-tsare mai tushe da tsari shine mafarin mafi yawan tsarin tsara; chunking na ma'ana yana da inganci mai kyau amma tsada.
- Nau'o'i kamar tebur, rubutun, da taɗi suna buƙatar dabarun kansu; Maida teburi zuwa rubutu na fili.
- Ƙara tushen/kwanaki/babi/ metadata na sirri da taken sashe zuwa kowace waƙa; Wannan shine tushen tacewa da ambato.
Aikin aikace-aikace
Rage wani sashe na takaddun da kuka zaɓa zuwa hanyoyi daban-daban guda uku: (1) ƙananan guda 200 alamomi, (2) matsakaita guda 500 (alamu 70), (3) manyan guda ɗaya. Yi tambayoyi guda 3 iri ɗaya don kowace dabara, yi alama da hannu wanne yanki don kawowa, kuma rubuta dalilin wace dabara ce ta fi dacewa da wannan takaddar. Sannan ƙara aƙalla filayen metadata huɗu da “ take” zuwa kowace waƙa. Idan takardar ta ƙunshi tebur, canza layin tebur zuwa rubutu na fili a cikin tsarin "filin: darajar".
jerin abubuwan dubawa
- [ ] Zan iya faɗi cewa ƙugiya ita ce mafi ƙanƙanta naúrar maidowa kuma girman shine ma'auni-ma'auni.
- [ ] Na san dalilin da ya sa haɗuwa ke hana asarar iyaka.
- [ ] Zan iya bambance tsakanin tushen sashi, ma'ana da sane da tsari.
- [ ] Zan iya daidaita dabarun don tebur, lamba da taɗi.
- [ ] Ina ƙarfafa dawowa ta ƙara metadata da taken babi ga kowace waƙa.