Markets & Operations

Data Miner 2 Release Notes

System Requirements

Release 19.10

Release Dates

Training: 10.1.2019
Production: 10.23.2019

About This Release

One New Feed and API Bug Fix.

New Features

New Feed

The following feed is being made available as part of FERC Order 844. October 2019 data for these feeds will be presented monthly from November 20th onwards. Note that data prior to October 2019 is still considered confidential and therefore there will be no data back-loaded for this feed.

  • Operator Initiated Commitments - This new feed will have economic max and reason for the Operator Initiated Commitments by zone as required by FERC 844.

Bug Fixes and Steps to Recreate


Sorting on DateTime Beginning UTC - Previously the sorting functionality on DateTime Beginning UTC parameter was not working as designed. With this fix, the sorting will work as designed on UTC dates.

Previous Releases


Release 19.07

Release Date
7.17.2019

About this release
Nine new feeds and bug fix for wind generation feed.

New Features
New Feeds
As noted in Release 19.01 Release Notes, the following feeds are being made available as part of FERC Order 844. July 2019 data for these feeds will be presented monthly from August 10th onwards. Note that data prior to July 2019 is still considered confidential and therefore there will be no data back-loaded for these feeds.
Daily Uplift by Zone - This feed will be retired and will be replaced by new feed called Uplift Credits by Zone. The new feed will have substantively the same information but the Credit Type pivoted into one single column. The Daily Uplift by Zone feed will be moved to the new ‘Retired’ category from ‘Generation’. This change was made to be consistent with the formatting of new data that will be required by FERC Order 844. While PJM tries to avoid substantially changing the structure of existing feeds, it was felt that in the long term, improved consistency in the uplift related feeds would be of benefit to users.
Uplift Charges by Zone - This new feed will have zonal uplift charge data as required by FERC 844.
Generator Specific Uplift Credits - This new feed will have generator specific uplift credits on a monthly basis as required by FERC Order 844 and is posted on a one month lag. Data for July 2019 will not appear until the September posting.
Load Response Uplift Credits - This new feed will have demand response specific uplift credits on a monthly basis as required by FERC Order 844 and is posted on a one month lag. Data for July 2019 will not appear until the September posting.
The following feeds will present the data that’s currently available on the Operational data page. Data layout changes have been made to this data to make the layout easier to understand and maintain, as well as reducing the amount of code changes needed for members in the event that changes are made in the future. Various sections of the Operational Data page have been split into 5 different feeds. Data for these feeds will be available from July 17 onwards.
Unverified Five Minute LMPs (Category: Locational Marginal Prices) - This feed contains the unverified real-time LMP data for a subset of pnodes which are zonal, aggregate, interface, hub and 500 Kv bus.
Transfer Interface Information (Category: System Information) - This feed contains the actual flow MW and transfer limits information for the Transfer Interfaces.
Instantaneous Load (Category: Load) - This feed contains the load MW for four regions. Loads are calculated from raw telemetry data and are approximate. The displayed values are NOT official PJM Loads.
Transmission Limits (Category: System Information) - This feed contains the current PJM transmission limits. This feed will be populated with data only when there are transmission constraints for that five minute interval.
Instantaneous Dispatch Rates (Category: System Information) - This feed contains the PJM instantaneous dispatch rates. The following dispatch rate information includes both the PJM electronic dispatch signals and manual dispatch signals. The electronic rates are updated every 15 seconds. The manual rates are updated with a frequency of 1 to 5 minutes and reflect individual generator dispatch. This data is provided for informational purposes ONLY and should not be relied upon by any party for the actual billing values. Missing interval data will not be backfilled due to the nature of the data. This data will be retained for 15 days only.
Data Layout Transformation Details
Datetime UTC and EPT stamps have been added all the feeds 500 KV Bus Locational Marginal Prices are posted under Unverified Five Minute LMP data with type = 500 KV
Current PJM Transmission Limits data is presented in a tabular form rather than a text string
Data will be retained for 30 calendar days except dispatch rates, which will be retained for 15 days
Any data for missing periods will be populated at pre-determined frequency for all the feeds except the Instantaneous Dispatch Rates. Missing data will not be populated for this feed because of the data update frequency.
New Category for the Feeds
New category called ‘Uplift’ will be added to the left navigation pane. This enhancement is to provide better grouping of uplift-related data. The above listed feeds that contain Uplift charges and credits data will be present in ‘Uplift’ category.

Fixes
Wind Generation - Previously the value “Other” in region was not being presented for this feed. With this release, the value “Other” will be present in region when appropriate.

Release 19.06

Release Date
7.1.2019

New Features
New Feeds
Solar Generation -This feed provides the hourly solar generation amounts in PJM. The Other" area includes external units and other units where the original regional location is not known.
FTR Congestion LMPs -This feed contains the congestion locational marginal price for the FTR pnodes

Enhancements
Frequently Accessed Data – Previously when users clicked on any feed from Frequently Accessed Data area, the data used to open in new window. With this enhancement, the data will open in a new tab keeping user experience consistent with the Explore Data Set button.

Release 19.04

Release Date
4.17.2019

Enhancements
Explore Data Set on Data Miner 2 User Interface – Previously when users clicked on Explore Data Set, the data used to open in a new window. When user clicked on Explore Data Set for a different feed, the window refreshed with data for that specific feed. Based on feedback from users, enhancements have been made to exploring data sets. With these enhancements, when users click on Explore Data Set button, the data will now open in a new tab within the same browser window. When users navigate to a different feed and click Explore Data Set for that feed, the data opens in a new tab leaving the previous tab unaffected.
Please Note: users will have to perform a one-time clear browser cache to be able to reload the interface. This can be achieved by using Ctrl+F5 keys on most browsers.

Release 19.03

Release Date
3.14.2019

About This Release
A validation in Markets Gateway has been preventing some resources from updating their energy parameters (economic min/max, emergency min/max) in real time because of comparisons to the SR Max value, which is an hourly value locked 65 min before the operating hour. With this new release, this validation will no longer occur in the Markets Gateway UI or via XML. Instead, the validation will be performed downstream in the optimization engines.

New Features
The validation in Markets Gateway for the following rules: 1) SR Max = Eco Max ≤ Emergency Max for all resources, and 2) SR Max ≤ Eco Max ≤ Emergency Max for resources that have submitted a modification request, has been removed. If a parameter is submitted on any Markets Gateway screen that violates these rules, Markets Gateway will accept the submission but the optimization engines will use the correct value. Participants are encouraged to submit and update their parameters in a timely manner to reflect the actual capability of their resources.
New/modified XML: No changes are required for XML submission

Associated External Interface Specification Guide version: Revision 61

Release 19.01

Release Date
1.29.2019

About This Release
A validation in Markets Gateway has been preventing some resources from updating their energy parameters (economic min/max, emergency min/max) in real time because of comparisons to the SR Max value, which is an hourly value locked 65 min before the operating hour. With this new release, this validation will no longer occur in the Markets Gateway UI or via XML. Instead, the validation will be performed downstream in the optimization engines.

New Features
The validation in Markets Gateway for the following rules: 1) SR Max = Eco Max ≤ Emergency Max for all resources, and 2) SR Max ≤ Eco Max ≤ Emergency Max for resources that have submitted a modification request, has been removed. If a parameter is submitted on any Markets Gateway screen that violates these rules, Markets Gateway will accept the submission but the optimization engines will use the correct value. Participants are encouraged to submit and update their parameters in a timely manner to reflect the actual capability of their resources.
New/modified XML: No changes are required for XML submission

Associated External Interface Specification Guide version: Revision 61

Release 18.12

Release Date
12.21.2018

About This Release
A validation in Markets Gateway has been preventing some resources from updating their energy parameters (economic min/max, emergency min/max) in real time because of comparisons to the SR Max value, which is an hourly value locked 65 min before the operating hour. With this new release, this validation will no longer occur in the Markets Gateway UI or via XML. Instead, the validation will be performed downstream in the optimization engines.

New Features
The validation in Markets Gateway for the following rules: 1) SR Max = Eco Max ≤ Emergency Max for all resources, and 2) SR Max ≤ Eco Max ≤ Emergency Max for resources that have submitted a modification request, has been removed. If a parameter is submitted on any Markets Gateway screen that violates these rules, Markets Gateway will accept the submission but the optimization engines will use the correct value. Participants are encouraged to submit and update their parameters in a timely manner to reflect the actual capability of their resources.
New/modified XML: No changes are required for XML submission

Associated External Interface Specification Guide version: Revision 61

Release 18.11

Release Date
11.30.2018

About This Release
A validation in Markets Gateway has been preventing some resources from updating their energy parameters (economic min/max, emergency min/max) in real time because of comparisons to the SR Max value, which is an hourly value locked 65 min before the operating hour. With this new release, this validation will no longer occur in the Markets Gateway UI or via XML. Instead, the validation will be performed downstream in the optimization engines.

New Features
The validation in Markets Gateway for the following rules: 1) SR Max = Eco Max ≤ Emergency Max for all resources, and 2) SR Max ≤ Eco Max ≤ Emergency Max for resources that have submitted a modification request, has been removed. If a parameter is submitted on any Markets Gateway screen that violates these rules, Markets Gateway will accept the submission but the optimization engines will use the correct value. Participants are encouraged to submit and update their parameters in a timely manner to reflect the actual capability of their resources.
New/modified XML: No changes are required for XML submission

Associated External Interface Specification Guide version: Revision 61

Release 18.09

Release Date
9.24.2018

About This Release
A validation in Markets Gateway has been preventing some resources from updating their energy parameters (economic min/max, emergency min/max) in real time because of comparisons to the SR Max value, which is an hourly value locked 65 min before the operating hour. With this new release, this validation will no longer occur in the Markets Gateway UI or via XML. Instead, the validation will be performed downstream in the optimization engines.

New Features
The validation in Markets Gateway for the following rules: 1) SR Max = Eco Max ≤ Emergency Max for all resources, and 2) SR Max ≤ Eco Max ≤ Emergency Max for resources that have submitted a modification request, has been removed. If a parameter is submitted on any Markets Gateway screen that violates these rules, Markets Gateway will accept the submission but the optimization engines will use the correct value. Participants are encouraged to submit and update their parameters in a timely manner to reflect the actual capability of their resources.
New/modified XML: No changes are required for XML submission

Associated External Interface Specification Guide version: Revision 61

Release 18.07a

Release Date
8.2.2018

Fixes
Operations Summary Feeds – Previously the peak statistics data showed discrepancies between the csv file posted on pjm.com and Data Miner 2. The issues originate from upstream data source discrepancies. This fix will ensure the load forecast, scheduled tie flow and internal scheduled capacity will match between both data sources.
Operations Summary - Projected RTO Statistics at Peak - Enhancements are made to improve “Scheduled Tie Flow Total” field description.
Operations Summary - Projected Scheduled Tie Flow – Enhancements are made to improve “Scheduled Tie Flow” field description.
Fixed Weighted Average Aggregate Definitions – Previously the data showed discrepancies between the csv file posted on pjm.com and Data Miner 2. Upon further investigation, the issue was identified to be a missed posting update on pjm.com No code changes are made to Data Miner 2 at this time.

Release 18.07

Release Date
7.2.2018

New Features
New feeds – These feeds will be available from 7/2 onwards
Day-Ahead Temperature Sets
First Available – This is a new field that will be available for all feeds from 7/2 onwards on the feed definition page. This field will show the earliest date data is available for that specific feed.

Enhancements

Real-Time Scheduled Interchange – Enhancements made to the user interface for this feed to add a top level filter on the “Datetime Beginning EPT” column
Historical Load Forecasts – Enhancements made to the user interface for this feed to add a top level filter on the “Datetime Beginning EPT” column
Operations Summary - Projected Area Statistics at Peak – Enhancements are made to improve Data Fields descriptions
Operations Summary - Projected RTO Statistics at Peak – Enhancements are made to improve Data Fields descriptions

Known Issues

This Data Miner 2 known issues section provides information on issues that impact users and are known to Data Miner 2 team.

Operations Summary Feeds - There is a known issue with the peak statistics data discrepancies between the csv file posted on pjm.com and Data Miner 2. The issues originate from upstream data source discrepancies and are actively being investigated. Update Posted on 7/20 – The issue with forecast data is identified as the sum of area forecasts are not matching the RTO forecast. This issue stems from individual area vs aggregated zones. PJM’s load forecasting system creates forecasts for a number of different geographic areas. It creates forecasts for individual zones as well as for larger aggregated areas (such as RTO and Mid Atlantic). Due to the nature of the neural net forecasting, these larger areas do not precisely match the sum of the individual component zones. Both forecasts are correct in that they are what was forecasted by the software, but we recognize the confusion caused by the difference in the values. To make the numbers more intuitive, PJM has decided to sum up the smaller areas to get the larger area forecast, rather than providing the neural net forecast for the larger territory for the Data Miner 2 posting. Internal scheduled capacity - The data posted in Data Miner 2 is not matching the data posted on pjm.com. Internal scheduled capacity - The data posted in Data Miner 2 is not matching the data posted on pjm.com. The known issue with Scheduled Tie Flow data on DM2 not matching pjm.com is being investigated at this time.

Fixed Weighted Average Aggregate Definitions - There is a known issue with the termination logic for the aggregate pnodes that is causing additional aggregate nodes to appear. This issue is actively being worked on.

Release 18.06

Release Date
5.31.2018

Fixes
Real-Time Scheduled Interchange Posting – Previously data for this posting was missing data for the last four hours of the day, and that data was not made available until the next day’s posting. This fix will ensure the data will be posted for all 24 hours every day.
Real-Time Five Minute LMPs – Previously, using datetime filters 20:00 to 23:00 was not returning any results. This fix will ensure this datetime filters will return data for the appropriate time block.

Enhancements
RSS Notifications – The RSS notifications for the following feeds will now include the Market Day the LMP data is posted for in the email body: a) Day-Ahead Hourly LMPs b) Real-Time Hourly LMPs c) Ancillary Service LMPs d) Real-Time Five Minute LMPs e) Ancillary Services Five Minute LMPs

Monthly Financial Transmission Rights Zonal LMPs – The pnode filter for this feed will be enhanced to a text box providing enhanced search capability on the user interface.

Off-Cost Operations – Enhancements made to the user interface for this feed to add a top level filter on the “Date” column.

Release 18.05

Release Date
4.30.2018

Fixes
Day Ahead Marginal Value and Constraints Posting - Enhancements have been implemented to reduce the time required to post the Day-Ahead Marginal Value and Constraints postings.
Market to Market Flowgate FFE – Enhancements have been implemented to move the posting time to 0830 EPT from 0900 EPT.
Settlements Verified Hourly LMPs – Updated the “Update Availability” description to “Monthly on the 4th business day at 10:00 a.m.” on the feed definition page. Note that this is a metadata update only that better reflects the actual posting time.
PJM Regulation Zone Preliminary Billing Data – The feed description has been updated to rename two columns - “Total PJM Regulation Market Capacity Clearing Price” and “Total PJM Regulation Market Performance Clearing Price” to “Total PJM Regulation Market Capacity Clearing Price Credits” and “Total PJM Regulation Market Performance Clearing Price Credits”.

Release 18.04

Release Date
3.29.2018

New Features
New feeds – These feeds will be available from 4/2 onwards:
Real-Time Five Minute LMPs
Ancillary Services Five Minute LMPs
Settlements Verified Five Minute LMPs
Settlements Verified Five Minute Ancillary LMPs
Daily Energy Market Generation Offers

Update to existing feeds – These updates to the existing feeds will be available from 4/2 onwards:
Real Time Marginal Value - As part of five minute settlements initiative, the shadow prices will be posted at a five minute granular level. Historic data will remain hourly.
Real Time Transmission Constraints – This feed will be retired as the data is now included in the Real Time Marginal Value feed. No new data will be posted to this feed. Historic data will remain available.
PJM Regulation Zone Preliminary Billing Data – As part of five minute settlements initiative, two new columns (Total PJM Regulation Market Capacity Clearing Price and Total PJM Regulation Market Performance Clearing Price) were added to the feed. Historic data for these new columns will be BLANK.
RTO Synchronized Reserve Preliminary Billing Data – As part of five minute settlements initiative, one new column (Total Subzone Tier 2 PJM-Scheduled MWh) are added to the feed. Historic data for this new column will be BLANK.
Day-Ahead LMP Posting: The posting of the Day-Ahead LMPs will now synchronize with other PJM Tools

Real-Time LMPs Posting: Enhancements are done to reduce the time required to post RealTime LMPs

Performance enhancements to five minute LMP data downloads - As part of the five minute settlements initiative, PJM has enhanced the performance of Data Miner 2. PJM has performed load testing of Data Miner 2. When 50 users simultaneously download the full prior day of five minute LMPs for all pnodes, it takes an average of 4.5 minutes to complete. This test case represents the expected typical scenario. The same test with a more extreme case of 250 simultaneous users yields download times of approximately 12 minutes. Naturally these times will vary based on system usage, the speed of your internet connection and other factors. These times are not a guarantee, but should help users set their expectations. Data Miner 2 uses caching to more quickly provide the most recent day of data; users querying older historical data should expect it to take longer.

UI Enhancement - The left navigation area where users select the feeds was previously not sorted. With this enhancement, the feeds within each category will be alphabetically sorted.

Fixes
Operations Summary Feed schedule - Previously the operations summary feeds were running at 8 AM to post only the 8 AM data. This scheduler is being modified to post Operations Summary data every hour starting from 5 AM to 8 AM. The 8 AM data will only be retained. This matches the functionality for the Operations Summary on pjm.com.
Date Alias “This Month” - Previously the date alias “ThisMonth” returned results that did not include the last four hours of the month. This fix will ensure “thismonth” will return results for the entire datetime within the calendar month.
Pnode Selector not working in Internet Explorer Browser - Previously pnode selector was not returning any results when users tried to select pnodes on LMP feeds using Internet Explorer. This fix will ensure the pnode selector will work as desired.
Inconsistency in the zone assigned to Pnodes in Data Miner 2's LMP output - Previously some pnodes in the LMP historic data were showing zones as AES instead of AEP. This fix will correct the zones for the relevant pnodes.

Release 18.03

Release Date
3.29.2018

New Features
New feeds – These feeds will be available from 4/2 onwards Real-Time Five Minute LMPs Ancillary Services Five Minute LMPs Settlements Verified Five Minute LMPs Settlements Verified Five Minute Ancillary LMPs Daily Energy Market Generation Offers Production: April 24, 2019
Update to existing feeds – These updates to the existing feeds will be available from 4/2 onwards
Real Time Marginal Value - As part of five minute settlements initiative, the shadow prices will be posted at a five minute granular level. Historic data will remain hourly. Real Time Transmission Constraints – This feed will be retired as the data is now included in the Real Time Marginal Value feed. No new data will be posted to this feed. Historic data will remain available. PJM Regulation Zone Preliminary Billing Data – As part of five minute settlements initiative, two new columns (Total PJM Regulation Market Capacity Clearing Price and Total PJM Regulation Market Performance Clearing Price) were added to the feed. Historic data for these new columns will be BLANK RTO Synchronized Reserve Preliminary Billing Data – As part of five minute settlements initiative, one new column (Total Subzone Tier 2 PJM-Scheduled MWh) are added to the feed. Historic data for this new column will be BLANK
Day-Ahead LMP Posting: The posting of the Day-Ahead LMPs will now synchronize with other PJM Tools
Real-Time LMPs Posting : Enhancements are done to reduce the time required to post RealTime LMPs
Performance enhancements to five minute LMP data downloads As part of the five minute settlements initiative, PJM has enhanced the performance of Data Miner 2. PJM has performed load testing of Data Miner 2. When 50 users simultaneously download the full prior day of five minute LMPs for all pnodes, it takes an average of 4.5 minutes to complete. This test case represents the expected typical scenario. The same test with a more extreme case of 250 simultaneous users yields download times of approximately 12 minutes. Naturally these times will vary based on system usage, the speed of your internet connection and other factors. These times are not a guarantee, but should help users set their expectations. Data Miner 2 uses caching to more quickly provide the most recent day of data; users querying older historical data should expect it to take longer.
UI Enhancement The left navigation area where users select the feeds was previously not sorted. With this enhancement, the feeds within each category will be alphabetically sorted.

Fixes:
Operations Summary Feed schedule Previously the operations summary feeds were running at 8 AM to post only the 8 AM data. This scheduler is being modified to post Operations Summary data every hour starting from 5 AM to 8 AM. The 8 AM data will only be retained. This matches the functionality for the Operations Summary on pjm.com. Date Alias “ThisMonth” Previously the date alias “ThisMonth” returned results that did not include the last four hours of the month. This fix will ensure “thismonth” will return results for the entire datetime within the calendar month. Pnode Selector not working in Internet Explorer Browser Previously pnode selector was not returning any results when users tried to select pnodes on LMP feeds using Internet Explorer. This fix will ensure the pnode selector will work as desired. Inconsistency in the zone assigned to Pnodes in Data Miner 2's LMP output Previously some pnodes in the LMP historic data were showing zones as AES instead of AEP. This fix will correct the zones for the relevant pnodes A validation in Markets Gateway has been preventing some resources from updating their energy parameters (economic min/max, emergency min/max) in real time because of comparisons to the SR Max value, which is an hourly value locked 65 min before the operating hour. With this new release, this validation will no longer occur in the Markets Gateway UI or via XML. Instead, the validation will be performed downstream in the optimization engines.

New Features
The validation in Markets Gateway for the following rules: 1) SR Max = Eco Max ≤ Emergency Max for all resources, and 2) SR Max ≤ Eco Max ≤ Emergency Max for resources that have submitted a modification request, has been removed. If a parameter is submitted on any Markets Gateway screen that violates these rules, Markets Gateway will accept the submission but the optimization engines will use the correct value. Participants are encouraged to submit and update their parameters in a timely manner to reflect the actual capability of their resources.
New/modified XML: No changes are required for XML submission

Associated External Interface Specification Guide version: Revision 61

Release 18.03

Release Date
2.15.2018

Fixes
API Portal Documentation Update Previously the attribute descriptions on the API management portal had generic descriptions. Based on user feedback, in this release, these descriptions have been enhanced and will be published on the API portal. This same information will also be published in the API Guide available on pjm.com

Release 18.02

Release Date
2.1.2018

Pnode termination date logic
Previously the pnodes that should have one long, continuous active period instead have several contiguous periods. This issue was being caused by how the pnode feed was being updated based on LMP data for that pnode. This fix will ensure that all active pnodes will have a continuous active period. Existing data will remain unchanged however this fix will take effect for all data going forward
Daily Generation Capacity missing data
A small number of missing intervals for the months of December 2017 and January 2018 were identified for the Daily Generation Capacity Feed. These data gaps were corrected.
Date Alias “Last Week” is incorrect [API Only]
Previously the Date Alias “Last Week” was not bringing back a full 7 days’ worth of data. This fix ensures that this date alias works as intended. The date alias “Last Week” will now return data where the date falls within the prior calendar week. This fix is implemented for all date aliases

Release 18.01

Release Date
1.11.2018

New Features
Feed Update Notifications: This enhancement provides users with a RSS Notification they may subscribe to. Once subscribed, the RSS feed will provide notifications whenever a feed is updated. Users will need to subscribe to each feed separately. This feature has been added based on member feedback.
Metadata for Real-Time and Day-Ahead LMP Reposts: This enhancement provides users with details on LMP reposts, similar to what is available on pjm.com today. The details will contain the specific dates and the pnode ids LMPs were reposted for. Notices will be posted every time there is a LMP repost by PJM. On the UI, this information will be displayed on the feed definition page. On the API, it will be available via the feed_metadata API response.

Fixes
Datetime Filters on UI and API Previously, the hour 00:00 was being considered as hour 23:00 for the Datetime Start and End filters on EPT and UTC dates. When users entered the hour 00:00 as the End date, 24 hours of data is being returned instead of just the hour 00:00. The EPT hour 19:00 was also being read as EPT hour 23:00 because of a defect in EPT to UTC conversion. This fix will resolve these issues and few other usability issues with the datetime filters. Please refer to the following discussion topic posted on Tech Change Community page for specifics on the datetime filters behavior after the release: https://pjm.force.com/techchangeforum/s/question/0D50V00003ghJkrSAE/
Reset Button on the UI Previously, clicking on Reset Button on the UI completely erases the datetime filters. This fix will ensure that the reset button will now bring back the default filters that appear upon page load.
Duplicate records in some feeds appear sporadically There is a processing bug that is casing duplicate records to appear in specific feeds. Enhanced logging has been added to provide further diagnostic information to investigate this issue so that it may be addressed in the future.