We have seen many apps in Android in which we will detect the object present in the image whether it may be any object. In this article, we will take a look at the implementation of image labeling in Android using Firebase ML Kit.
What we are going to build in this article?
We will be building a simple application in which we will be capturing an image of any object and from that, we will detect the objects present inside our image with the accuracy level. A sample video is given below to get an idea about what we are going to do in this article. Note that we are going to implement this project using the Java language.
Step by Step Implementation
Step 1: Create a New Project
To create a new project in Android Studio please refer to How to Create/Start a New Project in Android Studio. Note that select Java as the programming language.
Step 2: Connect your app to Firebase
After creating a new project in Android Studio connect your app to Firebase. For connecting your app to firebase. Navigate to Tools on the top bar. After that click on Firebase. A new window will open on the right side. Inside that window click on Firebase ML and then click on Use Firebase ML kit in Android. You can see the option below screenshot.
After clicking on this option on the next screen click on Connect to Firebase option to connect your app to Firebase.
Step 3: Adding dependency for language translation to build.gradle file
Navigate to the Gradle Scripts > build.gradle(Module:app) and add the below dependency in the dependencies section.
// dependency for firebase core.
implementation’com.google.firebase:firebase-core:15.0.2′
// Firebase ML dependency
implementation ‘com.google.firebase:firebase-ml-vision:24.0.3’
implementation ‘com.google.firebase:firebase-ml-vision-image-label-model:20.0.1’
Step 4: Adding permissions to access the Internet and meta-data in your Android Apps AndroidManifest file
Navigate to the app > AndroidManifest.xml file and add the below code to it. Comments are added in the code to get to know in more detail.
XML
<!-- below line is use to add camera feature in our app --><uses-feature android:name="android.hardware.camera" android:required="true" /> <!-- permission for internet --><uses-permission android:name="android.permission.INTERNET" /> |
Add the below line inside your application tag.
XML
<meta-data android:name="com.google.firebase.ml.vision.DEPENDENCIES" android:value="label" /> |
Below is the complete code for the AndroidManifest.xml file.
XML
<?xml version="1.0" encoding="utf-8"?> package="com.example.labelimage"> <!-- below line is use to add camera feature in our app --> <uses-feature android:name="android.hardware.camera" android:required="true" /> <!-- permission for internet --> <uses-permission android:name="android.permission.INTERNET" /> <application android:allowBackup="true" android:icon="@mipmap/ic_launcher" android:label="@string/app_name" android:roundIcon="@mipmap/ic_launcher_round" android:supportsRtl="true" android:theme="@style/Theme.LabelImage"> <activity android:name=".MainActivity"> <intent-filter> <action android:name="android.intent.action.MAIN" /> <category android:name="android.intent.category.LAUNCHER" /> </intent-filter> </activity> <meta-data android:name="com.google.firebase.ml.vision.DEPENDENCIES" android:value="label" /> </application> </manifest> |
Step 5: Working with the activity_main.xml file
Navigate to the app > res > layout > activity_main.xml and add the below code to that file. Below is the code for the activity_main.xml file.
XML
<?xml version="1.0" encoding="utf-8"?><RelativeLayout android:layout_width="match_parent" android:layout_height="match_parent" tools:context=".MainActivity"> <!--image view for displaying our image--> <ImageView android:id="@+id/image" android:layout_width="match_parent" android:layout_height="300dp" android:layout_alignParentTop="true" android:layout_centerHorizontal="true" android:layout_marginTop="10dp" android:scaleType="centerCrop" /> <LinearLayout android:id="@+id/idLLButtons" android:layout_width="match_parent" android:layout_height="wrap_content" android:layout_below="@id/image" android:orientation="horizontal"> <!--button for capturing our image--> <Button android:id="@+id/snapbtn" android:layout_width="0dp" android:layout_height="wrap_content" android:layout_margin="5dp" android:layout_marginTop="30dp" android:layout_weight="1" android:text="SNAP" android:textSize="25dp" android:textStyle="bold" /> <!--button for detecting the objects--> <Button android:id="@+id/labelBtn" android:layout_width="0dp" android:layout_height="wrap_content" android:layout_margin="5dp" android:layout_marginTop="30dp" android:layout_weight="1" android:text="Label" android:textSize="25dp" android:textStyle="bold" /> </LinearLayout> <!--recycler view for displaying the list of objects--> <androidx.recyclerview.widget.RecyclerView android:id="@+id/idRVResults" android:layout_width="match_parent" android:layout_height="wrap_content" android:layout_below="@id/idLLButtons" /> </RelativeLayout> |
Step 6: Creating a modal class for storing our data
Navigate to the app > java > your app’s package name > Right-click on it > New > Java class and name your class as DataModal and add the below code to it. Comments are added inside the code to understand the code in more detail.
Java
public class DataModal { // variables for our // string and confidence. private String result; private float confidence; // constructor public DataModal(String result, float confidence) { this.result = result; this.confidence = confidence; } // getter and setter methods public float getConfidence() { return confidence; } public void setConfidence(float confidence) { this.confidence = confidence; } public String getResult() { return result; } public void setResult(String result) { this.result = result; } } |
Step 7: Creating a layout file for displaying our recycler view items
Navigate to the app > res > layout > Right-click on it > New > layout resource file and name it as result_rv_item and add below code to it.
XML
<?xml version="1.0" encoding="utf-8"?><androidx.cardview.widget.CardView android:layout_width="match_parent" android:layout_height="wrap_content" android:layout_margin="5dp" android:elevation="8dp" app:cardCornerRadius="8dp"> <RelativeLayout android:layout_width="match_parent" android:layout_height="wrap_content" android:layout_margin="3dp"> <!--text view for our result--> <TextView android:id="@+id/idTVResult" android:layout_width="match_parent" android:layout_height="wrap_content" android:padding="3dp" android:text="Result" android:textColor="@color/black" /> <!--text view for our confidence--> <TextView android:id="@+id/idTVConfidence" android:layout_width="match_parent" android:layout_height="wrap_content" android:layout_below="@id/idTVResult" android:padding="3dp" android:text="Confidence" android:textColor="@color/black" /> </RelativeLayout> </androidx.cardview.widget.CardView> |
Step 8: Creating an adapter class for our RecyclerView
Navigate to the app > java > your app’s package name > Right-click on it > New > Java class and name it as resultRVAdapter and add the below code to it. Comments are added inside the code to understand the code in more detail.
Java
import android.content.Context;import android.view.LayoutInflater;import android.view.View;import android.view.ViewGroup;import android.widget.TextView; import androidx.annotation.NonNull;import androidx.recyclerview.widget.RecyclerView; import java.util.ArrayList; public class resultRVAdapter extends RecyclerView.Adapter<resultRVAdapter.ViewHolder> { // arraylist for storing our data and context private ArrayList<DataModal> dataModalArrayList; private Context context; // constructor for our variables public resultRVAdapter(ArrayList<DataModal> dataModalArrayList, Context context) { this.dataModalArrayList = dataModalArrayList; this.context = context; } @NonNull @Override public ViewHolder onCreateViewHolder(@NonNull ViewGroup parent, int viewType) { // inside on create view holder method we are inflating our layout file which we created. View view = LayoutInflater.from(parent.getContext()).inflate(R.layout.result_rv_item, parent, false); return new ViewHolder(view); } @Override public void onBindViewHolder(@NonNull ViewHolder holder, int position) { // inside on bind view holder method we are setting // data to each item of recycler view. DataModal modal = dataModalArrayList.get(position); holder.resultTV.setText(modal.getResult()); holder.confidenceTV.setText("" + modal.getConfidence()); } @Override public int getItemCount() { // returning the size of array list. return dataModalArrayList.size(); } public class ViewHolder extends RecyclerView.ViewHolder { // creating variables for our text view. private TextView resultTV, confidenceTV; public ViewHolder(@NonNull View itemView) { super(itemView); // initializing our views with their ids. resultTV = itemView.findViewById(R.id.idTVResult); confidenceTV = itemView.findViewById(R.id.idTVConfidence); } }} |
Step 9: Working with the MainActivity.java file
Go to the MainActivity.java file and refer to the following code. Below is the code for the MainActivity.java file. Comments are added inside the code to understand the code in more detail.
Java
import android.content.Intent;import android.graphics.Bitmap;import android.os.Bundle;import android.provider.MediaStore;import android.view.View;import android.widget.Button;import android.widget.ImageView;import android.widget.Toast; import androidx.annotation.NonNull;import androidx.appcompat.app.AppCompatActivity;import androidx.recyclerview.widget.LinearLayoutManager;import androidx.recyclerview.widget.RecyclerView; import com.google.android.gms.tasks.OnFailureListener;import com.google.android.gms.tasks.OnSuccessListener;import com.google.firebase.ml.vision.FirebaseVision;import com.google.firebase.ml.vision.common.FirebaseVisionImage;import com.google.firebase.ml.vision.label.FirebaseVisionImageLabel;import com.google.firebase.ml.vision.label.FirebaseVisionImageLabeler; import java.util.ArrayList;import java.util.List; public class MainActivity extends AppCompatActivity { // variables for our image view, image bitmap, // buttons, recycler view, adapter and array list. private ImageView img; private Button snap, labelBtn; private Bitmap imageBitmap; private RecyclerView resultRV; private resultRVAdapter resultRvAdapter; private ArrayList<DataModal> dataModalArrayList; @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); // initializing all our variables for views img = (ImageView) findViewById(R.id.image); snap = (Button) findViewById(R.id.snapbtn); labelBtn = findViewById(R.id.labelBtn); resultRV = findViewById(R.id.idRVResults); // initializing our array list dataModalArrayList = new ArrayList<>(); // initializing our adapter class. resultRvAdapter = new resultRVAdapter(dataModalArrayList, MainActivity.this); // layout manager for our recycler view. LinearLayoutManager manager = new LinearLayoutManager(MainActivity.this); // on below line we are setting layout manager // and adapter to our recycler view. resultRV.setLayoutManager(manager); resultRV.setAdapter(resultRvAdapter); // adding on click listener for our label button. labelBtn.setOnClickListener(new View.OnClickListener() { @Override public void onClick(View v) { // calling method // to label images. labelImage(); } }); // adding on click listener for our snap button. snap.setOnClickListener(new View.OnClickListener() { @Override public void onClick(View v) { // calling a method // to capture an image. dispatchTakePictureIntent(); } }); } static final int REQUEST_IMAGE_CAPTURE = 1; private void dispatchTakePictureIntent() { // inside this method we are calling an implicit intent to capture an image. Intent takePictureIntent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE); if (takePictureIntent.resolveActivity(getPackageManager()) != null) { // calling a start activity for result when image is captured. startActivityForResult(takePictureIntent, REQUEST_IMAGE_CAPTURE); } } @Override protected void onActivityResult(int requestCode, int resultCode, Intent data) { super.onActivityResult(requestCode, resultCode, data); // inside on activity result method we are setting // our image to our image view from bitmap. if (requestCode == REQUEST_IMAGE_CAPTURE && resultCode == RESULT_OK) { Bundle extras = data.getExtras(); imageBitmap = (Bitmap) extras.get("data"); // on below line we are setting our // bitmap to our image view. img.setImageBitmap(imageBitmap); } } private void labelImage() { // inside the label image method we are calling a // firebase vision image and passing our image bitmap to it. FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(imageBitmap); // on below line we are creating a labeler for our image bitmap // and creating a variable for our firebase vision image labeler. FirebaseVisionImageLabeler labeler = FirebaseVision.getInstance().getOnDeviceImageLabeler(); // calling a method to process an image and adding on success listener method to it. labeler.processImage(image).addOnSuccessListener(new OnSuccessListener<List<FirebaseVisionImageLabel>>() { @Override public void onSuccess(List<FirebaseVisionImageLabel> firebaseVisionImageLabels) { // inside on success method we are running a loop to get the data from our list. for (FirebaseVisionImageLabel label : firebaseVisionImageLabels) { // on below line we are getting text from our list. String text = label.getText(); // on below line we are getting its entity id String entityId = label.getEntityId(); // on below line we are getting the // confidence level of our modal. float confidence = label.getConfidence(); // after getting all data we are passing it to our array list. dataModalArrayList.add(new DataModal(text, confidence)); // after adding a new data we are notifying // our adapter that data has been updated. resultRvAdapter.notifyDataSetChanged(); } } }).addOnFailureListener(new OnFailureListener() { @Override public void onFailure(@NonNull Exception e) { // error handling for on failure method Toast.makeText(MainActivity.this, "Fail to get data..", Toast.LENGTH_SHORT).show(); } }); }} |
Now run your app and see the output of the app.
Output:
Note: If you are facing the following NDK at ~/Library/Android/sdk/ndk-bundle did not have a source.properties file error then please refer to this article.
