Riba:
- Ikon gane maɓuɓɓukan bayanai daban-daban (database, API, file, scraping na yanar gizo) da maƙasudin kowane kuma fahimtar tsarin daidai.
- Ƙarfin yin ƙima mai maimaitawa ta hanyar kimanta ko samfurin yana wakiltar yawan jama'a da zaɓin son rai
- Ikon kawar da zubar da bayanai a matakin tattarawa da kiyaye iyakokin doka/da'a ta hanyar yin tambayar 'Zan samu shi a lokacin tsinkaya' a kowane shafi?
Kowane bincike yana da kyau kamar ingancin bayanan da kuke tattarawa. Hatta samfurin da ya fi ci gaba a duniya zai haifar da sakamako mara inganci idan ya yi aiki da bayanan da aka tattara ba daidai ba, an ƙirƙira su da son zuciya, ko kuma sun ƙunshi bayanai game da gaba. A kimiyyar kwamfuta, an taƙaita wannan ƙa'idar a matsayin "datti a ciki, datti" (sharar cikin, datti). A cikin wannan rukunin, za mu rufe lokacin tattara bayanai: fahimtar tushen, yin samfuri, yin tambayoyi masu inganci, da faɗakarwa ga haɗarin yaɗuwar bayanai daga rana ɗaya. Hankali na wucin gadi taimako ne mai ƙarfi a wannan matakin; Yana rubuta tambayar SQL, yana taƙaita daftarin aiki API, tsara kwangilar bayanai. Amma dan Adam ne ke yanke shawarar irin bayanan da kuke tattara da kuma ko bayanan suna wakiltar ku.
Sanin tushen bayanai
Bayanai sun zo daga wurare daban-daban, kuma kowane tushe yana da nasa ramukan. Database (tsararrun bayanan da aka adana a cikin tebur, yawanci ana tambayar su tare da SQL) shine tushen da ya fi kowa; Abin dogara ne, amma wajibi ne a fahimci makircinsa da kyau. API (Application Programming Interface) yana ba da bayanan kai tsaye amma yana ɗaukar haɗarin iyakoki na sauri da canje-canjen tsari. Fayilolin (CSV, Excel, JSON) suna da sassauƙa amma suna iya yin rashin daidaituwa. Rushewar yanar gizo yana da ƙarfi, amma yana da iyakokin doka da ɗabi'a; Ba kowane rukunin yanar gizo ba ne za a iya gogewa.
Hankali: Don gogewar yanar gizo da tattara bayanai ta atomatik, bi sharuɗɗan amfani da shafin, fayil robots.txt da KVKK/GDPR. Tarin bayanai mara izini yana haifar da alhaki na doka. A cikin mahallin tsaro na bayanai, yi amfani da kayan aikin tattara bayanai kawai akan tsarin da aka ba ku izini kuma don dalilai na tsaro/bincike; An hana shiga mara izini ko gogewa.
Fahimtar tsarin: fahimtar da bayanai
Kafin tattara saitin bayanai, dole ne ku fahimci tsarinsa (sunayen ginshiƙan, nau'ikan bayanansu, ma'anarsu, da alaƙar su da juna). AI yana da fa'ida sosai anan wajen ƙirƙirar "kamus ɗin bayanai" - tebur yana bayanin abin da kowane shafi yake nufi. Amma bayanin da AI ke samarwa shine tsinkaya; Tabbatar da ainihin ma'anar kowane shafi tare da ƙungiyar da ta samar da bayanan. Misali, ginshiƙi mai suna "halaye" na iya ƙunsar 0/1/2; Ƙungiyoyin da suka samo asali ne kawai suka san ko waɗannan "ana jiran/an yarda/an soke" ko wani abu dabam.
Tebur mai zuwa yana taƙaita nau'ikan albarkatu na asali da kuma taka tsantsan:
Source
karfi batu
tarko
Yadda AI ke taimakawa
Bayanan Bayani na SQL
Tsarin tsari, abin dogaro
Complex JOINs
Yana rubuta daftarin tambaya
API
live data
Iyakar gudu, canza siffa
Takaitattun takardu, lambar ja
CSV/Excel
M, mai sauri
Tsarin rashin daidaituwa
Karanta/fasa lambar
gogewar yanar gizo
Faɗin isa
Iyakar doka/da'a
Fassarar daftarin aiki (a cikin hukuma)
Shiga/bayanan taron
daki-daki
girma girma
Tace tambaya
Misali: shin sashin yana wakiltar duka?
Yawancin lokaci, kuna aiki tare da samfurin (wani ɓangaren da aka zaɓa daga yawan jama'a) maimakon dukan bayanai. Tambaya mai mahimmanci ita ce: shin wannan samfurin yana wakiltar yawan jama'a? Bambancin zaɓi shine tarkon da ya fi yawa. Misali, idan kawai samfurin masu amfani daga wayar hannu, ba za ku ga masu amfani da yanar gizo ba kuma sakamakonku zai zama yaudara. Samfuran bazuwar (kowane rikodin yana da daidai damar zaɓe) shine mafi aminci a mafi yawan lokuta; amma a cikin jerin bayanai na lokaci, ana yin rarrabuwa bisa ga tsarin lokaci maimakon dazu (za mu ga wannan a cikin Raka'a 7 da 10).
Leak sani daga rana daya
Zubar da bayanai shine tushen mafi yawan masifu kuma yawanci yana tasowa yayin lokacin tattara bayanai. Misali: lokacin tsinkayar "an soke shi", idan kun ƙara ginshiƙin "cancellation date" zuwa bayanan, ƙirar tana duban gaba. A lokacin taron, yi tambaya ɗaya ga kowane shafi: "Shin da gaske zan sami wannan bayanin a lokacin da na yi hasashen?" Idan amsar ita ce a'a, wannan ginshiƙi yana yoyo. Za mu yi cikakken bayani kan wannan batu a cikin Raka'a 10; Amma ya kamata a fara wayar da kan jama'a daga rana ta farko.
uku mini lokuta
Case 1 - Matsalar wakilci. Banki ɗaya ya tattara bayanai kawai akan lamunin da aka amince don ƙirar haɗarin bashi (rubutun 18,500). Babu kin amincewa a cikin bayanan. Samfurin ya kasance ba daidai ba a duniyar gaske saboda bai taɓa ganin yadda waɗanda aka ƙi zasu yi ba. Darasi: Samfurin ya kamata ya zama wakilin dukan jama'ar da kuke yanke shawara.
Case 2 - Silent form canza. Wata ƙungiya tana jan bayanan farashi daga API kowace rana. Wata rana, mai bada API ya canza kuɗin daga USD zuwa EUR, amma sunan yankin ya kasance iri ɗaya. An tattara bayanai a cikin kuskuren naúrar na kwanaki 12; Layuka 3,200 sun lalace. Darasi: A kai a kai duba girma da daidaiton tsari a cikin bayanan API.
Case 3 - Farkon yabo. Wani manazarci ya haɗa da ginshiƙin "dalilin rufe asusun" lokacin tattara bayanai don kimanta "churn". An cika wannan shafi ne kawai bayan abokin ciniki ya tafi. Samfurin ya samar da daidaiton 97% akan gwajin gwajin; Bai yi aiki a samarwa ba saboda wannan shafi ba komai a lokacin tsinkaya. Darasi: Yi wa kowane shafi tambayar "Shin ina da shi a lokacin hasashen?"
Samfura huɗu masu kwafi
1) Cire ƙamus na bayanai:
Matsayinku: mataimakin masanin kimiyyar bayanai. A ƙasa akwai sunayen ginshiƙan da samfurin (maras sani) ƙimar tebur. Ga kowane ginshiƙi, jera kimanta ma'anarsa, nau'in bayanai, da yuwuwar haɗarin inganci a cikin tebur. Alama ginshiƙan da ba ku da tabbas a matsayin "tabbatar da ake buƙata"; ma'ana yin.Shafukan: [manna a nan]
2) Lambar samfur (bazuwar, mai maimaitawa):
Ina da pandas df. Rubuta lambar da ke fitar da wakilci 5% samfurin bazuwar daga layuka 200,000. Yi amfani da random_state=42 (don sake haifuwa). Ƙara lamba don bincika cewa rarraba ajin samfurin yayi kama da yawan jama'a.
3) Tambayar dubawa:
Zan ba ku wannan jerin ginshiƙai. Burina shine in hango "an soke" (0/1). Ga kowane ginshiƙi, kimanta ko a zahiri zan sami shi a lokacin hasashen kuma yi masa alama a matsayin "aminci / m / leak". Rubuta dalilin ku a cikin jumla ɗaya. Rukunin: [jeri]
4) SQL ja daftarin tambayar:
Ina da tebur "oda" da "abokan ciniki" a cikin PostgreSQL. Rubuta tambayar JOIN wanda ya haɗa umarni na kwanaki 90 na ƙarshe tare da birnin abokin ciniki kuma ya dawo da jimlar adadin da adadin umarni kowane birni. Yi bayanin tace kwanan wata da yadda ake sarrafa biranen NULL. Zan gudanar da tambaya kuma in tabbatar da ita.
Rauni mai ƙarfi / Ƙarfi mai ƙarfi
Rawanin faɗakarwa:
Zamo mani kyakkyawan samfurin bayanai daga wannan bayanan.
"mai kyau" ba shi da tabbas; Wanne zane, wane lokaci, wane girman, wane dalili bai bayyana ba. AI kawai za ta samar da jigon, mai yuwuwar tambayar kuskure.
Ƙarfi mai ƙarfi:
Matsayinku: Mataimakin SQL. Ina da tebur "ma'amaloli": ginshiƙai id, customer_id, date (timestamp), adadin (lambobi), tashar (rubutu: 'web'/'mobile'). Aiki: Rubuta tambaya mai maimaitawa (ƙididdigewa tare da ORDER BY) wanda ke dawo da layukan wakilai 10,000 daga kowane tashoshi na shekara ta 2024. Manufa: nazarin kwatancen tashar. Lissafin zato na tambayar ku.
Anan tebur, manufa, girman da maimaitawa a bayyane suke.
Kuskuren gama gari
- Ba tambayar wakilcin samfurin ba. Bayani mai sauƙin isa ba daidaitattun bayanai ba ne; son rai na zaɓi yana karkatar da sakamakon.
- Daidaita ma'anar ginshiƙi zuwa AI. Ƙungiyar tushen ta san ma'anar; Kada ku yi amfani da hasashen AI ba tare da tabbatar da shi ba.
- Ba bin tsarin API/canjin naúrar ba. Canjin shiru yana tattara gurbatattun bayanai na kwanaki.
- Yin watsi da ɗigon ruwa a matakin tarin. Idan tambaya "Shin ina da shi a lokacin tsinkaya" ba a yi shi da wuri ba, samfurin zai ba da nasara na ƙarya.
- Tattara bayanan da ba izini ko doka ba. Cin zarafin robots.txt, sharuɗɗan amfani da KVKK babban haɗari ne.
Tukwici: Ajiye "katin bayanai" mai shafi ɗaya don kowane sabon tushen bayanai: tushe, kwanan wata, adadin layuka, sanannun iyakoki, da ginshiƙai cikin haɗarin yabo. Wannan katin yana adana tambayar "menene wannan bayanan" da sake fasalin watanni bayan haka.
A takaice
An iyakance ingancin bincike ta hanyar ingancin bayanan da aka tattara. Sanin tushen (database, API, file, scrape) da makirci da kyau; tabbatar da samfurin wakiltar yawan jama'a; Kawar da yabo daga rana ɗaya ta hanyar tambayar kowane shafi "Shin ina da shi a lokacin tsinkaya?" AI babban mai haɓakawa ne don tambaya da aikin daftarin aiki, amma mutane suna yanke shawarar abin da bayanai za su tattara da wakilcin sa. Iyaka na iko, doka da sirri koyaushe suna zuwa na farko.
Aikin aikace-aikace
Zaɓi tushen bayanai (daga kasuwancin ku ko hasashe). Sami daftarin ƙamus na bayanai daga AI tare da samfurin “harar ƙamus na bayanai” a sama; Sannan auna kowane shafi da hannu don ganin ko ya zube. Yi ƙoƙarin nemo aƙalla ginshiƙi masu tuhuma/yaɗi kuma rubuta a cikin jumla ɗaya dalilin da yasa yake da haɗari.
jerin abubuwan dubawa
- [ ] Na tabbatar da tushen bayanai da tsarin tare da ƙungiyar tushen?
- [ ] Na duba cewa samfurin wakilcin jama'a ne?
- [ ] Na tambayi kowane shafi tambayar "zan samu shi a lokacin kimantawa?"
- [ ] Na yi samfurin maimaituwa (tsayayyen iri)?
- [ ] Shin na bincika iyakokin tattarawa na doka/da'a (iko, robots.txt, KVKK)?