Gains:
- Ability to recognize the structure, units and typical quality problems of SCADA, smart meters and market data
- Ability to detect missing data, outliers and timestamp problems with artificial intelligence and establish a cleaning workflow
- Ability to verify traps such as resolution, time zone and daylight saving time in energy time series in AI output
Almost all energy analysis starts with a time series: measurements arranged along the time axis. The 15-minute consumption of a meter, the second rotation speed of a turbine, the hourly current of a feeder (a distribution line)—all are time series. In this unit, we will learn where energy data comes from, what pitfalls it contains, and how to use artificial intelligence to make this data reliable. Let's establish this principle from the beginning: even the smartest analysis with dirty data gives a dirty result. In energy engineering, most of the time is spent not on the model, but on making the data reliable, and artificial intelligence provides the safest contribution in this part.
Sources and Structure of Energy Data
Energy data comes primarily from several sources. SCADA (supervisory control and data acquisition system) collects instantaneous measurements from switchboard and grid equipment; It is usually in seconds or minutes resolution and includes quantities such as voltage, current, power and temperature. Smart meter data typically measures consumption at 15-minute intervals and is the basis for billing and demand analysis. Meteorological data (temperature, irradiance, wind speed) is the input of production and demand forecasting. Market data (hourly prices) is the input of economic analysis.
The common feature of these data is that they contain a timestamp (date-time tag to which each measurement belongs) and one or more measurement values. This is the most critical point: if the timestamp is not understood correctly, everything else collapses.
Resolution, Time Zone and Daylight Saving Time Traps
A surprisingly large portion of the errors in energy data come from the time axis. Let's highlight three traps in particular.
Resolution confusion. Does a meter report 15-minute energy (kWh) or instantaneous power (kW)? Adding them up gives different results: the average of the four kW values for 15 minutes is the power, while the sum of the four does not correspond to the hourly energy (the sum of the quarter-hour energies is required). No summation can be made without understanding the unit and resolution together.
Time zone. Is the data kept in local time (TRT/UTC+3 for Türkiye) or universal time (UTC)? Two series coming from different systems may be in different time zones; The three-hour shift when combining disrupts the entire peak hour analysis.
Daylight saving time (DST). In countries that switch to daylight saving time, an hour "disappears" once a year and "repeats" once a year. This creates 23 or 25 hour days on those days and breaks hourly profiles. Since Türkiye has implemented permanent daylight saving time (UTC+3) since 2016, this problem does not appear on local data, but you still encounter it when working with international or old data.
Notice: Is the label for the timestamp "beginning of range" or "end of range"? Does a record labeled 14:00 indicate 14:00-14:15 or 13:45-14:00? This single decision could shift the peak clock by one slice. Make this rule clear at the beginning for each data set.
Step by Step: Data Preparation Workflow with Artificial Intelligence
Step 1 — Discovery. Know the data: how many rows, what range, what resolution, what unit? Giving the AI the first few hundred rows and summarizing the structure quickly reveals column meanings and possible units. But AI's unit estimate is a hypothesis; It is confirmed from the source document.
Step 2 — Standardize the timeline. Snap to a single time zone, fix resolution, specify tag rule (beginning/ending). AI can generate code that detects irregular intervals and missing timestamps.
Step 3 — Incomplete and duplicate records. Communication breakdowns create emptiness and double recording again. AI produces a set of rules that flag gaps and duplications; short gaps (e.g. a single 15 minutes) can be filled by a suitable method, long gaps are excluded from the analysis and reported.
Step 4 — Outliers. Negative consumption (no production), power exceeding the physical upper limit, sudden jumps. AI offers statistical and rule-based outlier detection; But an outlier is not always an error — it can also be a real event (for example, a factory going online). The engineer makes this distinction.
Step 5 — Verification. Cleaned data is tested with physical anchors: sunrise/sunset must reset solar generation, night load must be lower than daytime, total energy must be consistent with the bill.
Three Mini Cases: By the Numbers
Case 1 — One hour slip. One analyst's solar production data showed the noon peak at 14:00 instead of 13:00. The problem wasn't with the panel; The data was UTC but it was thought to be local time. When the three-hour shift was noticed (the appearance of production before sunrise gave away the anchor) the whole analysis improved; Investment decisions were prevented based on misinterpretation.
Case 2 — Duplicate recordings. In one distribution feeder's 30-day hourly data, total consumption was approximately 4 percent higher than expected. AI-assisted replay detection showed that some hours were recorded twice in communication reconnections. After clearing 68 repeat records the total matched the bill.
Case 3 — False zeros. One meter was reporting consumption as "0" during a communication loss; However, consumption continued. These false zeros lowered the average and misled the demand forecast. AI suggested the rule "Inspect 0 values together with the presence flag"; false zeros were marked as missing data and separated from true zeros (e.g., closed workplace).
Weak Prompt / Strong Prompt
Weak prompt:
Clear this counter data.[data]
Powerful prompt:
Your role: Energy data analyst. The following meter data is for 15 minute intervals, local time (UTC+3), the timestamp indicates the STARTING of the interval, unit kWh.Tasks:1) List missing timestamps and duplicate records (number and time).2) Mark separately negative value and values exceeding the physical upper limit (contract power 25 kW); don't delete, just mark.3) Offer a check to distinguish false zeros from real zeros that are suspected of miscommunication.Don't fill in any values by yourself; Identify it first and I will confirm the filling.
Powerful prompt gives resolution, time zone, tag rule and upper limit from the start; It imposes a “detect first, ask later” discipline on AI. This way the data will not be silently corrupted.
Four Copiable Templates
1) Data profiling:
Create a profile of the following data: number of rows, date range, resolution, estimated unit and meaning of each column, missing rate, min/max/average. Mark unit estimates with the "needs to be verified" tag.
2) Time axis control:
In this time series, detect: (a) gaps out of expected range, (b) repeating timestamps, (c) possible shifts due to daylight saving time/time zone. List each finding with its time range. Write your correction suggestion separately; APPLICATION.
3) Outlier rule:
Mark outliers using these physical limits: power 0-[X] kW, temperature [A]-[B] °C. Also make a separate list of statistical outliers (e.g. deviating too far from the median). For each outlier, raise the question "possible error or actual event"; Let me decide.
4) Verification after cleaning:
Test the cleaned data with the following physical anchors and report inconsistencies: solar production should be zero at night; total energy must be within the expected bill level; The night load must be lower than the daytime peak. Write pass/fail for each anchor.
Data Quality Issues Table
problem
symptom
verification anchor
Time zone shift
The top clock is in an illogical place
Align with sunrise/sunset
Summer time (DST)
23/25 hour days
Count day length
fake zero
0 in case of communication loss
Match with communication flag
re-register
Above total bill
Check timestamp uniqueness
Unit confusion
Rank 60/1000 times perverted
Verify kW↔kWh conversion
negative value
Negative consumption when there is no production
Physical sign rule
Common mistakes
- Silently filling in missing data with zero. Zero means "no consumption", not "no data"; mixing the two distorts the average and forecast.
- Automatically delete outlier. An outlier may be a real event; Engineer review required before deletion.
- Assuming time zone. Saying "it's probably local" is one of the most expensive mistakes; Always verify from the source.
- Collecting the solubility by mixing. Adding power (kW) values as energy (kWh) gives an incorrect total.
- Not verifying cleanliness. Cleaned data may also be corrupt; Be sure to test it with physical anchors.
Tip: Write down each step of data cleaning in a “data log”: how many records were marked, how many were populated, which rule was used. This log ensures both reproducibility and auditability; a year later "where did this number come from?" answers the question.
In summary
The basis of energy analysis is a clean time series. Data comes from SCADA, meter, meteorology and market; each has resolution, unit, time zone, and daylight saving time traps. AI is very powerful in accelerating missing/duplicate/outlier detection, but deletion and filling decisions should remain with the engineer, and the result should be validated with physical anchors. No analysis is reliable without a proper understanding of the timestamp.
Application task
Take an energy time series (meter, production or temperature) in your hand. Request a quality report from the AI using the “Timeline audit” and “Data profiling” templates. Then manually verify three things: are the unit and resolution correct, is the peak clock physically reasonable, is the total energy consistent with the bill? Briefly describe a data quality issue you found and how you fixed it.
checklist
- [ ] I have confirmed the source, resolution and unit of the data
- [ ] I clarified the timezone and timestamp label rule (beginning/ending)
- [ ] I checked the risk of 23/25 hour days due to summer time
- [ ] I detected missing and duplicate records and reported them
- [ ] I examined the outliers with the distinction of "error or event"
- [ ] I distinguished false zeros from real zeros
- [ ] I verified the cleaned data with physical anchors and kept a data log