Riba:
- Ability don bincika rarrabawa, tsakiya, yadawa da kuma rashin daidaituwa na masu canji tare da jadawali masu dacewa (histogram, filin akwatin, watsawa).
- Ƙarfin fassarar daidaituwa daidai kuma karanta shi tare da ɓarna mai ɓarna, ba tare da rikitar da shi tare da dalili ba
- Ƙarfin fahimtar cewa ƙididdiga taƙaice na iya zama yaudara (darasin Anscombe) da haɓaka hasashe waɗanda ke ƙayyade alkiblar ƙirar ƙira.
Kafin gina samfurin, wajibi ne a san bayanan. Kamar yadda likita ba ya rubuta magani ba tare da bincikar majiyyaci ba, masanin kimiyyar bayanai ba ya gina samfurin ba tare da "nazarin" bayanan ba. Ana kiran wannan jarrabawar bincike na bayanai (EDA a takaice): lokaci ne na gano rarrabawa, dangantaka, abubuwan mamaki da tarko na bayanan a gani da kuma a hoto. Manufar EDA ita ce ta haifar da hasashe, kama kurakurai, da ƙayyade alkiblar ƙira. Ilimin wucin gadi shine mataimaki mai ƙarfi sosai a wannan matakin: yana nuna wace ginshiƙi ya dace, ya rubuta lambar sa, yana fassara rarraba. Amma mutum yana yanke hukunci, tare da iliminsa/ta filin, shin tsarin da yake gani na gaske ne ko kuma kwatsam.
Menene EDA ke nema?
EDA na neman amsoshin tambayoyi hudu. Rarraba: Ta yaya ake rarraba kowane maɗaukaki-yana daidaitacce, karkatacciyar hanya, ko mafi girma biyu? Halin tsakiya da yadawa: Menene ma'anar, tsaka-tsaki, daidaitaccen karkata? Dangantaka: Ta yaya masu canji ke da alaƙa da juna? Abubuwan da ba a sani ba: Akwai dabi'un da ba a zata ba, gibi, rukuni? Waɗannan tambayoyi guda huɗu sun ƙayyade abin da fasalin zai yi aiki kuma wane samfurin ya dace.
Bari mu ayyana wasu mahimman ra'ayoyi. Histogram jadawali ne da ke raba kimar ma'auni na ma'auni zuwa tazara kuma yana nuna adadin abubuwan lura a cikin kowane tazara; Ita ce hanya mafi sauri don ganin rarrabawa. Skewness shine motsi na rarrabawa a daya hanya; Ana karkatar da sauye-sauye kamar samun kudin shiga zuwa dama (mafi yawan ƙasa, kaɗan ne masu girma). Makircin akwatin yana nuna tsaka-tsaki, quartiles da masu fita a kallo. Maɓallin watsawa yana nuna alaƙar masu canjin lambobi biyu tare da gajimare na maki.
Daidaitawa: akwai dangantaka amma a kula
Daidaitawa - ma'auni na yadda masu canji biyu ke tafiya tare; lamba tsakanin -1 da +1 - shine kayan aikin da aka fi amfani da su kuma mafi rashin fahimta na EDA. +1 yana nufin cikakkiyar haɓaka haɗin gwiwa, -1 yana nufin cikakkiyar alaƙar da ba ta dace ba, 0 yana nufin babu alaƙar layi. Akwai manyan tarko guda biyu. Na farko, sanannen ƙa'idar: daidaitawa ba dalili ba ne. Abubuwan shaƙatawa suna ƙaruwa tare da tallace-tallace na ice cream; ba don ɗayan yana kaiwa ga ɗayan ba, amma saboda lokacin rani (maɓallin ɓoye na uku) yana jawo duka biyun. Na biyu, daidaitawa kawai yana auna alaƙar linzamin kwamfuta; Yana iya nuna dangantaka mai ƙarfi amma curvilinear kusa da 0. Don haka lokacin kallon daidaitawa, tabbatar da duba maƙasudin watsawa kuma.
Tsanaki: Lokacin da AI ya ba da lambar haɗin kai, yana iya zamewa cikin harshe mai lalacewa kamar "A yana ƙaruwa B." Wannan yana da haɗari. Daidaitawa kawai yana nuna motsi tare; Kwarewa kawai, ilimin yanki, da zurfin tunani ne ke kafa dalili.
Anscombe quartet: me yasa ba kawai duba lambar ba
Akwai sanannen misali a cikin ƙididdiga: Anscombe quartet. Saitin bayanai daban-daban guda huɗu suna da kusan ma'ana iri ɗaya, bambance-bambance iri ɗaya, da daidaituwa iri ɗaya (0.82); Amma idan aka zana su, sun bambanta gaba ɗaya - madaidaiciyar layi ɗaya, ɗaya mai lanƙwasa, ɗaya gajimare wanda wani waje ɗaya ya ja. Darasin a bayyane yake: taƙaitaccen ƙididdiga na yaudara ne, kallon jadawali dole ne. AI na iya ba ku matsakaita da alaƙa, amma amincewa da waɗannan lambobin ba tare da ganin jadawali ba yana faɗuwa cikin tarkon Anscombe.
Teburin da ke ƙasa ya taƙaita wace ginshiƙi ya dace da wace tambaya:
Tambaya
dace mai hoto
Me ya nuna
Rarraba maɓalli guda ɗaya
Histogram / KDE
Siffai, skewness, adadin madaidaici
Center da outliers
Filin akwatin
Matsakaici, quartiles, outliers
Dangantakar lambobi biyu
watsa jadawalin
Jagoranci, ƙarfi, curvature
Kwatancen rukuni
tsarin bar
Bambanci tsakanin kungiyoyi
canza a kan lokaci
layin layi
Trend, yanayin yanayi
Alakar iri-iri
Taswirar zafi mai alaƙa
Matrix dangantaka gaba ɗaya
Paradox na Simpson: tattara bayanai na iya yin ƙarya
Wani tarko mai banƙyama a cikin EDA don sani shine Simpson's paradox: tsari na iya nuna hanya ɗaya lokacin da aka bincika dukkan bayanai tare, amma baya lokacin da aka raba bayanan zuwa ƙungiyoyi masu ma'ana. Misali na al'ada: gabaɗayan karɓar karɓa ga mata masu neman izini a jami'a ya bayyana ya yi ƙasa da na maza; Amma idan aka dubi sashe da sashe, yawan karbuwar mata ya fi maza a mafi yawan sassan. Daga ina? Mata sun nemi ƙarin gasa (ƙananan ƙimar karɓa); Haɗaɗɗen lambar tana adana wannan ɓoyayyiyar canjin. Darasin a bayyane yake: lokacin duban jimla ko matsakaita, tabbatar da bincika ko lambar tana ba da labari iri ɗaya lokacin da aka karkasu zuwa ƙungiyoyi masu ma'ana (banki, lokaci, tashar). Lokacin da AI ta ba ku ƙimar haɗin gwiwa, tambayar "har yanzu yana aiki lokacin da kuka raba shi" zai sami mafi yawan sakamako masu ɓarna tun da wuri.
uku mini lokuta
Case 1 - Tsaunuka masu ɓoye biyu. Wani manazarci ya gano matsakaicin shekarun abokin ciniki ya zama 41 kuma ya ruwaito shi a matsayin "taro mai matsakaicin shekaru." Amma histogram ya nuna kololuwa biyu: 22-28 da 55-62 shekaru kungiyoyin. Matsakaicin 41 ba su wakilci kowa a zahiri ba. Darasi: shirya rarraba kafin aminta da ma'anar.
Case 2 - Haɗin kai. Wata ƙungiya ta sami alaƙar 0.78 tsakanin "kashe talla" da "tallace-tallace" kuma ta ninka kasafin kuɗi. Duk da haka, abin da ya jawo duka biyun shine kamfen na yanayi; a lokacin yakin neman zabe, dangantakar ta ragu zuwa 0.10. 340 dubu TL ƙarin tallace-tallace bai haifar da dawowar da ake tsammani ba. Darasi: kuskuren daidaitawa don dalili yana da tsada.
Case na 3 - Layin da aka zana shi kaɗai. Binciken ƙasa ya nuna dangantaka mai ƙarfi tsakanin murabba'in murabba'in da farashi (r=0.80). Lokacin da aka zana makircin watsawa, kusan dukkanin dangantakar an halicce su ne ta hanyar alatu guda 12 miliyan TL; Lokacin da aka cire wannan batu, alaƙar ta ragu zuwa 0.31. Darasi: Koyaushe karanta alaƙar tare da ɓarna.
Samfura huɗu masu kwafi
1) Takaitawar EDA ta atomatik:
Ina da pandas df. Rubuta lambar da ke samar da: (1) df.info() da df.describe(), (2) histogram na kowane ginshiƙi na lambobi, (3) ƙidaya ga kowane ginshiƙi. Kar a yi sharhi, kawai samar da lambar ganowa; Ina fassara alamu.
2) Daidaituwa + na gani (tare da fa'idar causality):
Rubuta lambar da ke zana matrix mai daidaitawa da taswirar zafi don ginshiƙan lamba. Hakanan zana madaidaicin madaidaicin nau'i-nau'i guda 3 masu alaƙa. Ƙara layin sharhi zuwa abubuwan da ake fitarwa yana tunatar da ku cewa haɗin gwiwa ba ya nufin haifarwa. Kada ku yi da'awar dalili.
3) ganewar asali:
Don ginshiƙin "income_tl": rubuta lambar da ke samar da histogram, makircin akwatin, ƙimar skewness da tsaka-tsaki/ma'ana kwatanta. Zan fassara ko rarraba ya karkata ko a'a; kawai ba da code.
4) Kwatancen sashi:
Kwatanta abokan ciniki ta "tashar" (web/mobile): rubuta lambar da ke samar da akwatin akwatin matsakaicin adadin, matsakaicin adadin, adadin umarni, da adadin rarraba ga kowane tashoshi. Ba da shawarar wane gwajin ya dace don mahimmancin ƙididdiga na bambanci tsakanin tashoshi biyu, amma yanke shawara ya rage nawa.
Rauni mai ƙarfi / Ƙarfi mai ƙarfi
Rawanin faɗakarwa:
Nemo wani abu mai ban sha'awa a cikin wannan bayanan.
"Mai sha'awa" ba a bayyana shi ba; AI baya ganin bayanan, don haka ko dai ya sanya shi ko ya yi bayanai na gaba ɗaya. Babu alkibla, babu masu canji, babu manufa.
Ƙarfi mai ƙarfi:
Matsayinku: Mataimakin EDA. df ginshiƙai: shekaru (int), revenue_tl (float), birni (categori), tashar (web/mobile), adadin (float). Manufa: don gano abubuwan da suka shafi "yawan". Aiki: (1) lambar da ke tsara rarraba adadin, (2) jadawalai masu rarraba tsakanin adadin da shekaru da kuma samun kudin shiga_tl, (3) adadin adadin adadin a cikin tashar tashar. Tabbatar da da'awar dalili; Kawai samar da lambar ganowa kuma zan fassara tsarin.
Anan, manufar, masu canji da iyakar "da'awar dalili" sun bayyana a sarari.
Kuskuren gama gari
- Dogaro da ƙididdiga taƙaice kawai. Kamar yadda Quartet na Anscombe ya nuna, ma'anar guda ɗaya tana ɓoye rabe-rabe daban-daban; duba jadawalin.
- Kuskuren daidaitawa don dalili. Haɗin kai baya nuna cewa ɗayan yana kaiwa ga ɗayan; Hattara da ɓoyayyun masu canji.
- Kallon daidaitawa ba tare da ɓarna ba. Daya daga waje ko curvilinearity yana ɓatar da lambar.
- Taƙaitaccen rarraba mai kololuwa/skewed biyu tare da ma'ana. Matsakaicin ƙila ba zai wakilci kowa ba.
- Tsallake EDA kuma tafi kai tsaye zuwa ƙirar. Bayanan da ba a gane ba yana nufin samfurin makaho.
Tukwici: Tare da kowane sabon saitin bayanai, ciyar da mintuna 30 na farko kawai zana hotuna: histogram na kowane lamba, watsar mahimman mahimmin nau'i-nau'i, ginshiƙi na rukuni. Waɗannan mintuna 30 ɗin suna hana kurakuran ƙirar ƙira na gaba don kwanaki masu zuwa.
A takaice
EDA shine lokaci na sanin bayanai kafin gina samfuri kuma yana neman amsoshin tambayoyi hudu: rarrabawa, yada cibiyar, dangantaka da abubuwan da ba su da kyau. Ƙididdiga taƙaice na iya zama yaudara; kallon jadawali dole ne (Anscombe lecture). Haɗin kai kayan aiki ne mai ƙarfi, amma dalili ba shine kuma yakamata a karanta shi tare da makircin watsawa. AI babban mai haɓakawa ne don ba da shawarar zane-zane da lambar rubutu; Amma mutane suna fassara tare da ilimin filin su ko tsarin da suke gani na gaske ne ko kuma daidaituwa.
Aikin aikace-aikace
Zaɓi saitin bayanai kuma kawai yi EDA: Cire tarihin ƙididdiga na aƙalla mabanbantan lambobi uku, ƙayyadaddun makirci na nau'i-nau'i biyu na masu canji, da taswirar zafi mai daidaitawa. Nemo aƙalla “mamaki” ɗaya (kamar rarraba tare da kololuwa biyu, ɗan takara don alaƙar zage-zage, dangantakar da ke jawo hankalin wani abu ɗaya) kuma bayyana shi a cikin jumla ɗaya. Rubuta ba tare da yin wani dalili ba.
jerin abubuwan dubawa
- [ ] Shin na tsara yadda ake rarraba kowane ma'auni na lambobi?
- [ ] Na duba alaƙar da ke tattare da ɓarna?
- [ ] Shin na kiyaye dalilai da alaƙa daban?
- [ ] Shin, ban lura da karkatacce ko rarraba-biyu ba kuma na amince da ma'anar a makance?
- [ ] Shin na samo aƙalla fannin ƙirar ƙira ɗaya / hasashe daga binciken EDA na?