Episode 531: Automatic Search on TV Listings

Monday, April 17, 2017

Is information collection a "business"?

There seems to be various opinions. If you watching television programs in the conference room, it might be a "business".

And if you are working for a department store, you may care about news concerning "Department stores". Or if you work for an Internet company, you might be curious about keywords such as 'AI', 'IoT', and 'big-data'.

If "special program" related to those keywords is scheduled as a television broadcast, you can't miss it. Particularly if it is planned by public broadcasting station, the impact on society could be very big.

＜NHK API control screen＞

NHK API that opened to anyone

NHK, a public broadcasting station in Japan, has published "Program Guide API". In June 2016, they started provision of Ver.2 , and it is now possible to acquire program information of "7 days ahead".

Response excerpt

"id" : "2017041215654",
"event_id" : "15654",
"start_time" : "2017-04-12T04:30:00+09:00",
"end_time" : "2017-04-12T05:00:00+09:00",
"title" : "ＮＨＫニュース　おはよう日本",
"subtitle" : "▼国内外の最新ニュース　▼スポーツ情報　▼地域の課題や話題のリポート　▼日本と世界の気象情報",
"content" : "※番組内容は変更になる場合があります　▼番組ＨＰ　ｈｔｔｐ：／／ｗｗｗ．ｎｈｋ．ｏｒ．ｊｐ／ｏｈａｙｏｕ／",
"act" : "【キャスター】森田洋平，【気象キャスター】酒井千佳",


This API is a very simple OPEN API.

As long as you register yourself, anyone can access it immediately, and regarding usage rules, it is very clear that "up to 300 accesses a day".
There is no authorization flow such as OAuth1 / OAuth2, it is not even secure communication (HTTPS)!

On the other hand, it can be said that it is APIs strict to standard specifications.

That is, if you are using various APIs on a daily basis, you would suppose that the characters that can be used on the website (UTF - 8) are communicated as they are. However, it is "Truly NHK". They firmly comply to the rule which "Send and receive with only alphanumeric characters (ASCII characters)" (Unicode escape) (JSON RFC7159) , so each and every Japanese character is converted to \nXXXX. (Even though It is hard to use in verifying communication log etc.)

[Keyword Search on TV Listing]

Automation of "work"

This Workflow App automatically obtains program list information through the NHK API. And it automatically search whether there is no program matching the preset keywords.

For example, if you set the keyword "Kyoto", you will be able to confirm in the email that there is "Information program for foreigners" at tomorrow morning at 4 o'clock. It is a different approach from "Web search" as typified by so-called "news applications" and "Google Alert".

Although you will understand when you look closely at the business flow, it will explore under the same conditions 24 hours later, even if you neglect the result.

Setting Example

  • TERRESTRIAL TV General channel: Kyoto, Gion, Maiko
  • SATELLITE TV BS 1 channel: Kyoto, Gion, Maiko, Hanamachi, Ponto-cho, Kaburenjo

Notification email excerpt

2017-04-14 04:11:00
◇ＯＮＥ　ＭＩＮＵＴＥ　ＡＤＶＩＳＥＲ「関西空港から駅への行き方」
あの有名外国人スターが来日！？関空で駅までの行き方が分からないというスターにＯＮＥ　ＭＩＮＵＴＥ　ＡＤＶＩＳＥＲが楽しく解説します。京都・嵐山で外国人旅行者に人気の隠れスポットも紹介。

[Keyword Search on TV Listing:"1. Enter search criteria" screen]

Setting method of Automatic step

The crux of this Workflow App (Business Process) is to search in program listings that has been automatically acquired.

That is, at the automatic step, it is necessary not only to "acquire data via API communication" but also "to extract necessary data from acquired data". If you have programming knowledge, it is a good idea to set server side JavaScript in "Script Task". (If you do not, you should use only the service step setting by utilizing the "Addon file of service step" in which the script is packaged.)

In the case of "Script Task", you can also deal with the tragedy such as;

for the keyword of "京都 (Kyoto)",  "東京都 (Tokyoto)" is hit.

In other words, you will be able to set "exclusion word", or to enable  "AND search", and enjoy various improvements according to the usage.

Incidentally, in the Cloud-based workflow "Questetra BPM Suite", there is a restriction of  "up to 1 million characters" as a storage limit for String type data. For example, if you try to get all the program listings of terrestrial and satellite broadcast at once (nhkService = "tv") and try to store it in String type data, you may get an error, so be careful. (It's a story of my experience...)

＜Setting example of Script Task "TV listing retrieval"＞
// TV Program search via NHK Program List API (ver. 20170410)
// (c) 2017, Questetra, Inc. (the MIT License)
// Info source: ＮＨＫ

//// == Config & Retrieving ==
// Your "Registered application"  for "Program Guide API"
// http://api-portal.nhk.or.jp/
var nhkKey = "Your-Key-32-Your-Key-32-Your-Key";

// http://api-portal.nhk.or.jp
var nhkArea = data.get( "q_nhkArea" ) + "";
var nhkService = data.get( "q_nhkService" ) + "";
// General"g1",Educational"e1",Satellite1"s1",Satellite Premium"s3"
var nhkDate = data.get( "q_nhkDate" ) + "";
var searchWordsCsv = data.get( "q_searchWordsCsv" ) + "";


//// == Calculating ==
var accessLog = "";
var matchLog = "";

/// -- GET program list --
var uri = "http://api.nhk.or.jp/v2/pg/list/";
uri += nhkArea + "/";
uri += nhkService + "/";
uri += nhkDate + ".json?key=";
uri += nhkKey;

// com.questetra.bpms.core.event.scripttask.HttpClientWrapper
// https://www.questetra.com/ja/tour/m2/m230/
// HttpRequestWrapper
var requestMsg = httpClient.begin();

// HttpResponseWrapper
var responseMsg = requestMsg.get( uri );
accessLog += "---GET--- " + responseMsg.getStatusCode() + " " + responseMsg.getCharset() + "\n";
accessLog += responseMsg.getResponseAsString() + "\n\n";

/// -- Word Search --
var responseObj = JSON.parse( responseMsg.getResponseAsString() );
var searchWords = searchWordsCsv.split(",");

var tmp = ""; 
for (var i = 0; i < responseObj.list[nhkService].length; i ++) { // Bracket notation 
  tmp = responseObj.list[nhkService][i].start_time.slice(11, 16) + "\n"; 
  tmp += responseObj.list[nhkService][i].title + "\n"; 
  tmp += responseObj.list[nhkService][i].subtitle + "\n"; 
  tmp += responseObj.list[nhkService][i].content + " "; 
  tmp += responseObj.list[nhkService][i].act + "\n"; 

  for (var j = 0; j < searchWords.length; j ++) { 
    if ( tmp.match( searchWords[j] ) ) { 
      matchLog += tmp + "\n+++ \n\n"; 
      break; 
    } 
  } 
} 

//// == Data Updating ==
retVal.put( "q_accessLog", accessLog );
retVal.put( "q_matchLog", matchLog );


＜Example of Servis Task Addon file＞
<?xml version="1.0" encoding="UTF-8"?><service-task-definition> 

<label>NHK Program Search</label> 

<configs> 
  <config name="conf_DataIdA" required="true" form-type="TEXTFIELD"> 
    <label>A: Set NHK KEY</label> 
  </config> 
  <config name="conf_DataIdB" required="true" form-type="SELECT" select-data-type="STRING_TEXTFIELD"> 
    <label>B: Select AREA CODE</label> 
  </config> 
  <config name="conf_DataIdC" required="true" form-type="SELECT" select-data-type="STRING_TEXTFIELD"> 
    <label>C: Select SERVICE CODE</label> 
  </config> 
  <config name="conf_DataIdD" required="true" form-type="SELECT" select-data-type="DATE"> 
    <label>D: Select DATE</label> 
  </config> 
  <config name="conf_DataIdE" required="true" form-type="SELECT" select-data-type="STRING_TEXTFIELD"> 
    <label>E: Set SEARCH WORDS CSV</label> 
  </config> 
  <config name="conf_DataIdF" required="true" form-type="SELECT" select-data-type="TEXTAREA"> 
    <label>F: Select Access Log</label> 
  </config> 
  <config name="conf_DataIdG" required="true" form-type="SELECT" select-data-type="TEXTAREA"> 
    <label>G: Select Match Log</label> 
  </config> 
</configs> 


<script><![CDATA[ 
// TV Program search via NHK Program List API (ver. 20170411)
// (c) 2017, Questetra, Inc. (the MIT License)
// Info source: ＮＨＫ

//// == Config & Retrieving ==
// Your "Registered application"  for "Program Guide API"
// http://api-portal.nhk.or.jp/
var nhkKey = configs.get( "conf_DataIdA" ) + "";

// http://api-portal.nhk.or.jp
var nhkArea = data.get( configs.get( "conf_DataIdB" ) ) + "";
var nhkService = data.get( configs.get( "conf_DataIdC" ) ) + "";
 // General"g1",Educational"e1",Satellite1"s1",Satellite Premium"s3"
var nhkDate = data.get( configs.get( "conf_DataIdD" ) ) + "";
var searchWordsCsv = data.get( configs.get( "conf_DataIdE" ) ) + "";

var dataIdF = configs.get( "conf_DataIdF" );
var dataIdG = configs.get( "conf_DataIdG" );

//// == Calculating ==
var accessLog = "";
var matchLog = "";

/// -- GET program list --
var uri = "http://api.nhk.or.jp/v2/pg/list/";
uri += nhkArea + "/";
uri += nhkService + "/";
uri += nhkDate + ".json?key=";
uri += nhkKey;

// com.questetra.bpms.core.event.scripttask.HttpClientWrapper
// https://www.questetra.com/ja/tour/m2/m230/
// HttpRequestWrapper
var requestMsg = httpClient.begin();

// HttpResponseWrapper
var responseMsg = requestMsg.get( uri );
accessLog += "---GET--- " + responseMsg.getStatusCode() + " " + responseMsg.getCharset() + "\n";
accessLog += responseMsg.getResponseAsString() + "\n\n";

/// -- Word Search --
var responseObj = JSON.parse( responseMsg.getResponseAsString() );
var searchWords = searchWordsCsv.split(",");

var tmp = ""; 
for (var i = 0; i < responseObj.list[nhkService].length; i ++) { // Bracket notation 
  tmp = responseObj.list[nhkService][i].start_time.slice(11, 16) + "\n"; 
  tmp += responseObj.list[nhkService][i].title + "\n"; 
  tmp += responseObj.list[nhkService][i].subtitle + "\n"; 
  tmp += responseObj.list[nhkService][i].content + " "; 
  tmp += responseObj.list[nhkService][i].act + "\n"; 

  for (var j = 0; j < searchWords.length; j ++) { 
    if ( tmp.match( searchWords[j] ) ) { 
      matchLog += tmp + "\n+++ \n\n"; 
      break; 
    } 
  } 
} 


//// == Data Updating ==
retVal.put( dataIdF, accessLog );
retVal.put( dataIdG, matchLog );

]]></script> 
</service-task-definition>


＜SAMPLE RESPONCE＞ (* Partly edited)
{
  "list":{
    "g1":[
      {
        "id" : "2017041219033",
        "event_id" : "19033",
        "start_time" : "2017-04-12T04:11:00+09:00",
        "end_time" : "2017-04-12T04:15:00+09:00",
        "area":{
          "id" : "260",
          "name" : "京都"
        },
        "service":{
          "id" : "g1",
          "name" : "ＮＨＫ総合１",
          "logo_s":{
            "url" : "//www.nhk.or.jp/common/img/media/gtv-100x50.png",
            "width" : "100",
            "height" : "50"
          },
          "logo_m":{
            "url" : "//www.nhk.or.jp/common/img/media/gtv-200x100.png",
            "width" : "200",
            "height" : "100"
          },
          "logo_l":{
            "url" : "//www.nhk.or.jp/common/img/media/gtv-200x200.png",
            "width" : "200",
            "height" : "200"
          }
        },
        "title" : "ＮＨＫプレマップ「連続テレビ小説　ひよっこ」",
        "subtitle" : "",
        "content" : "",
        "act" : "",
        "genres":[
          "0207"
        ]
      },
      {
        "id" : "2017041215654",
        "event_id" : "15654",
        "start_time" : "2017-04-12T04:30:00+09:00",
        "end_time" : "2017-04-12T05:00:00+09:00",
        "area":{
          "id" : "260",
          "name" : "京都"
        },
        "service":{
          "id" : "g1",
          "name" : "ＮＨＫ総合１",
          "logo_s":{
            "url" : "//www.nhk.or.jp/common/img/media/gtv-100x50.png",
            "width" : "100",
            "height" : "50"
          },
          "logo_m":{
            "url" : "//www.nhk.or.jp/common/img/media/gtv-200x100.png",
            "width" : "200",
            "height" : "100"
          },
          "logo_l":{
            "url" : "//www.nhk.or.jp/common/img/media/gtv-200x200.png",
            "width" : "200",
            "height" : "200"
          }
        },
        "title" : "ＮＨＫニュース　おはよう日本",
        "subtitle" : "▼国内外の最新ニュース　▼スポーツ情報　▼地域の課題や話題のリポート　▼日本と世界の気象情報",
        "content" : "※番組内容は変更になる場合があります　▼番組ＨＰ　ｈｔｔｐ：／／ｗｗｗ．ｎｈｋ．ｏｒ．ｊｐ／ｏｈａｙｏｕ／",
        "act" : "【キャスター】森田洋平，【気象キャスター】酒井千佳",
        "genres":[
          "0000",
          "0002",
          "0100"
        ]
      },
    ]
  }
}


<Data Items List>


[Free Download]
<Similar Models>
<<Related Articles>>

[Japanese Entry (和文記事)]
Label: , , , ,