{"id":2437,"date":"2021-05-25T00:59:58","date_gmt":"2021-05-24T15:59:58","guid":{"rendered":"https:\/\/appinventor.tmsoftwareinc.com\/?p=2437"},"modified":"2021-05-26T11:00:36","modified_gmt":"2021-05-26T02:00:36","slug":"%e4%ba%ba%e5%b7%a5%e7%9f%a5%e8%83%bd%e7%94%bb%e5%83%8f%e5%88%86%e9%a1%9e%e3%82%a2%e3%83%97%e3%83%aa2","status":"publish","type":"post","link":"https:\/\/appinventor.tmsoftwareinc.com\/en\/archives\/2437","title":{"rendered":"Artificial intelligence Image Classification App 2"},"content":{"rendered":"

Let’s make an Image Classification App with higher accuracy without extension!<\/h2>\r\n

You can change the machine learning model easily, because this app does not use extension.<\/p>\r\n

\"\"<\/a>
Click to see the app in action!<\/figcaption><\/figure>\r\n

From the Project menu, select New Project and name it “WhatisThis2”.<\/span><\/p>\r\n

Why LookExtension extension is not needed?<\/h3>\r\n

With tensorflow.js, you can use machine learning models in a browser. The LookExtension extension developed at MIT uses tensorflow.js. However, it is not easy to change the model because the extension must be rebuilt to change the machine learning model. In addition, the model used in this extension is small in size, so the accuracy is not that good. To solve this problem, we have developed a method for image classification using machine learning models by simply using a web page that contains JavaScript to be used in the browser. This method uses an external website to load the image files on the web page, but the model can be easily modified by simply changing the JavaScript on the web page.<\/span><\/p>\r\n

Design editing<\/span><\/span><\/h3>\r\n

Layout and component placement<\/span><\/span><\/h4>\r\n

As shown in the figure below, place a WebView, a TextBox from the User Interface palette, and a HorizontalArrangement layout from the Layout palette. Drag and drop two buttons from the User Interface palette and rename them as the PictureButton and the InferenceButton, and place them in the HorizontalArrangement1. and place them in the layout. As invisible components, drag and drop a Camera from the Media palette and a Clock from the Sensors palette, and two Webs from the Connectivity palette. Rename one Web component to “upload” and the other to “removefile”. Rename the hint in TextBox 1 to “Result”, the text in the PictureButton to “Take Picture”, and the text in the InferenceButton to “Inference”.<\/p>\r\n

\"\"<\/a>
Click to enlarge<\/figcaption><\/figure>\r\n

Adding a web page that contains JavaScript<\/h4>\r\n

Create the following html file and add it to your assets by clicking the “Upload File” button on the Media. The file name should be index.html. The following is a brief description of this file.<\/p>\r\n

Line 3: Read tensowflow.js
Line 4: Read MobileNet
Line 10: Image tags for captured image files
Line 18-24: WebGL2 function
Line 27-30: setimage() function. Set the URL of the image file passed from the App Inventor app.
Line 33-44: classify() function. Make inferences and pass the results to the App Inventor app.
Line 47-62<\/span>: app() function. Determine if WebGL2 is present and set backEnd appropriately, then load the MobileNet model.<\/p>\r\n

\r\n
\r\n
<html>\r\n<head>\r\n<script src=\"https:\/\/cdn.jsdelivr.net\/npm\/@tensorflow\/tfjs\"><\/script>\r\n<script src=\"https:\/\/cdn.jsdelivr.net\/npm\/@tensorflow-models\/mobilenet\"><\/script>\r\n<\/head>\r\n<body>\r\n<div><input type=\"text\" id=\"output\" size=\"100\" value=\"\"><\/div>\r\n\r\n\r\n<img id=\"img\" crossorigin src=\"https:\/\/img.appinventor.tmsoftwareinc.com\/images\/tfjs.png\" width=\"80%\"\/>\r\n\r\n\r\n\r\n<script >\r\nlet net;\r\n\r\n\r\nvar supportsWebGL2 = ( function () {\r\ntry {\r\nreturn !! window.WebGLRenderingContext && !! document.createElement( 'canvas' ).getContext( 'webgl2' );\r\n} catch( e ) {\r\nreturn false;\r\n}\r\n} )();\r\n\r\n\r\nfunction setimage() {\r\nvar appInventorInput = window.AppInventor.getWebViewString();\r\ndocument.getElementById('img').src = appInventorInput;\r\n}\r\n\r\n\r\nasync function classify() {\r\nwindow.AppInventor.setWebViewString('');\r\n\r\n\r\nconst imgEl = document.getElementById('img');\r\n\r\n\r\nconst result = await net.classify(imgEl);\r\nconst ret = result[0]['className'];\r\n\/\/ document.getElementById(\"output\").value = ret;\r\nwindow.AppInventor.setWebViewString(ret);\r\n}\r\n\r\n\r\nasync function app() {\r\nif(supportsWebGL2){\r\ndocument.getElementById(\"output\").value = 'WebGL2 is supported';\r\n}else{\r\ndocument.getElementById(\"output\").value = 'WebGL2 is not supported';\r\ntf.setBackend('cpu');\r\n}\r\n\r\n\r\ndocument.getElementById(\"output\").value = 'Loading MobileNet..';\r\n\r\n\r\n\/\/ Read MobileNet\r\nnet = await mobilenet.load();\r\ndocument.getElementById(\"output\").value = 'Successfully loaded model';\r\n}\r\n\r\n\r\napp();\r\n<\/script>\r\n<\/body>\r\n<\/html><\/code><\/pre>\r\n<\/div>\r\n<\/div>\r\n
Programming using the block editing function<\/h3>\r\n
Global Variables<\/h4>\r\n
The first step is to define the global variables. Click on the variable in the Built-in block and drag and drop the “Initialize global name to” blocks to set it as shown below. production is true when building apk file and\u00a0 false when using MIT AI2 Companion<\/span>. imageHost is the webserver to upload the image files. waitTime is the time in milliseconds to wait to get the inference results. The other variables are temporary.<\/p>\r\n
\"\"<\/a>
Click to enlarge<\/figcaption><\/figure>\r\n
Screen Initialize event<\/h4>\r\n
Incorporate the following blocks into the “when Screen1 Initialize” block. The first three blocks set temporary global variables. They are, in order, the URL of the uploaded image file, the URL of the PHP script used to upload the image file, and the URL of the PHP script used to delete the image file. The next if block sets the URL of the WebViewer1 to a different path because the asset path is different when building the apk file and when running on MIT AI2 Companion. Finally, we set the wait time for the timer and disable the timer.<\/p>\r\n
\"\"<\/a>
Click to enlarge<\/figcaption><\/figure>\r\n
PictureButton click event<\/h4>\r\n
Incorporate a “TextBox1 Text” block with blank text, a “call Camera1 TakePictire” block, and a “InferenceButton Enabled” block with a false logic block into the “when PictureButton Click” block<\/span><\/span><\/p>\r\n
\"\"<\/a><\/p>\r\n
Camera1 AfterPicture event<\/h4>\r\n
In the “when Camera1 AfterPicture” block, set the global variable “photoPath” to the image taken by the Camera1, set the global variable “uploadURL” to the URL of the upload web, and call upload PostFile with the image to upload it. At the same time, activate the InferenceButton.<\/p>\r\n
\"\"<\/a><\/p>\r\n
Upload GotText event<\/h4>\r\n
In the “When upload GotText” block, if the upload is successful, the name of the uploaded file will be returned in the responseContent, so set the global variable imageName to this file name. Then set the full URL of the image file to the WebViewString in WebView1 and call the setimage() JavaScipt function. This will display the uploaded image in the WebView1.<\/p>\r\n
\"\"<\/a><\/p>\r\n
InferenceButton click event<\/h4>\r\n
In the “when InferenceButton Click” block, put the blank text in the “TextBox1 Text” block, and call the classify() JavaScipt function to perform the inference. At the same time, enable the timer.<\/span><\/p>\r\n
\"\"<\/a><\/p>\r\n
Timer event<\/span><\/span><\/h4>\r\n
It takes some time for the WebViewString of WebView1 to be returned after calling the classify() JavaScript function, so we use a timer to wait for a while. This time is set in the global variable waitTime, which is 500 milliseconds (0.5 seconds). In the “when Clock1 Timer” block, disable Clock1, display the WebViewString of WebView1 in the TextBox1 Text, set removeURL with imageName to the URL of the removefile web, and then call the removefile Get method to delete the file from the webserver.<\/p>\r\n
\"\"<\/a><\/p>\r\n
That’s it! Test it on your phone. The classification results will be displayed at the bottom.<\/div>\r\n
\r\n
\"\"<\/a>
Click to enlarge<\/figcaption><\/figure>\r\n<\/div>\r\n
\r\n
Download source<\/h4>\r\n\r\n\r\n Source code of this App is available at MIT App Inventor Gallery<\/a><\/div>\r\n
\r\n
PHP source codes<\/h4>\r\n
They are the PHP source codes used to upload and delete image files. You can refer to it when using your own web server. For browser security, web servers need to return Access-Control-Allow-Origin “*” in the Header. For Apache web server, set Header set Access-Control-Allow-Origin “*”.<\/p>\r\n
upload.php<\/p>\r\n
\r\n
<?php\r\n$data = file_get_contents('php:\/\/input');\r\n$tmpfname = tempnam(\"\/var\/www\/------\/images\", \"prefix\");\r\nif (!(file_put_contents($tmpfname,$data) === FALSE)) echo basename($tmpfname); \/\/ file could be empty\r\nelse echo \"File xfer failed.\";\r\n?><\/code><\/pre>\r\n<\/div>\r\n
remove.php<\/p>\r\n
\r\n
<?php\r\nif(isset($_GET['fn'])) {\r\n    $fname = \"\/var\/www\/------\/images\/\" . $_GET['fn'];;\r\n    if(file_exists($fname)) {\r\n        echo \"unlinked $fname\";\r\n        unlink($fname);\r\n    }\r\n}\r\n?><\/code><\/pre>\r\n<\/div>\r\n<\/div>","protected":false},"excerpt":{"rendered":"
Let’s make an Image Classification App with higher accuracy without extension! You can change the machine learning model easily, because this app does not use extension. From the Project menu, select New Project and name it “WhatisThis2”. Why LookExtension extension is not needed? With tensorflow.js, you can use machine learning models in a browser. The […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":"","_locale":"en_US","_original_post":"https:\/\/appinventor.tmsoftwareinc.com\/?p=2375"},"categories":[4],"tags":[],"_links":{"self":[{"href":"https:\/\/appinventor.tmsoftwareinc.com\/wp-json\/wp\/v2\/posts\/2437"}],"collection":[{"href":"https:\/\/appinventor.tmsoftwareinc.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/appinventor.tmsoftwareinc.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/appinventor.tmsoftwareinc.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/appinventor.tmsoftwareinc.com\/wp-json\/wp\/v2\/comments?post=2437"}],"version-history":[{"count":52,"href":"https:\/\/appinventor.tmsoftwareinc.com\/wp-json\/wp\/v2\/posts\/2437\/revisions"}],"predecessor-version":[{"id":2508,"href":"https:\/\/appinventor.tmsoftwareinc.com\/wp-json\/wp\/v2\/posts\/2437\/revisions\/2508"}],"wp:attachment":[{"href":"https:\/\/appinventor.tmsoftwareinc.com\/wp-json\/wp\/v2\/media?parent=2437"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/appinventor.tmsoftwareinc.com\/wp-json\/wp\/v2\/categories?post=2437"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/appinventor.tmsoftwareinc.com\/wp-json\/wp\/v2\/tags?post=2437"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}