Interaction Flow Library - Experimental version

v0.23.3-experimental-01 - Documentation last updated on 11/06/2020

The Interaction Flow Library is an Android library for Pepper the robot. It was developed using the QiSDK.

It provides high level APIs to help you, the developer, create a robotic application with a whole interaction flow. This flow suits the design of a human-robot interaction in a B2B environment.

The flow is divided into 5 different steps, which are customizable to suit the client application needs:

  • Initialization: loads your resources, localizes Pepper if needed, etc…
  • Idle: Pepper looks for humans.
  • Start: Pepper saw a human and will try to attract it.
  • Interaction: Pepper succeeded in engaging the human, he can now interact with it. When the interaction’s conditions aren’t met anymore, the interaction will end and Pepper will go back to idling.
  • Unavailable: troubleshooting step that occurs if something goes wrong.

The library handles all these steps seamlessly, providing default GUI & behavior all the time.

If you don’t want any specific GUI for a given step, the flow will display a default fragment showing an image of Pepper’s torso, aiming at lessen the visual impact of the tablet.

The bare minimum that you need to provide is the interaction’s content.

You can also customize each step of the flow, including the GUI and the behavior. This will be explained in the Customization part of this documentation.

Minimum configuration

  • Pepper 1.9
  • API level 6

Getting started

Add the Experimental Maven repository to your project’s build.gradle file:

allprojects {
    repositories {
        maven { url 'https://qisdk.softbankrobotics.com/experimental/maven' }
    }
}

Add the dependency to your module’s build.gradle file:

dependencies {
    implementation 'com.softbankrobotics:interaction-flow:0.23.3-experimental-01'
}

How to use

For the bare minimum, you just need to have these classes:

  • A behavior that implements the InteractionBehavior interface, which will contain your interaction logic.
  • A chosen activity that will be the one responsible for handling the interaction flow.

Behavior

Each step has its own behavior, and they all work the same way:

  • When the flow enters a step, this step’s behavior onEnter method will be called. Here is where you should play robotic actions.
  • When the flow exits a step, this step’s behavior onExit method will be called. Here, you have to cancel any robotic actions played in the onEnter method. This method MUST be synchronous.

The initialization & unavailable steps behaviors are a little different, they only have an onEnter method and no onExit method:

  • Initialization: the onEnter method MUST be synchronous. When it ends, the flow will continue to the next step. If it throws, the flow will go to the Unavailable step.
  • Unavailable: there is no way to exit this step for now, so there is no need for an onExit method.

Interaction behavior implementation

The interaction behavior is pretty straight-forward, it just needs to implement 2 methods:

  • onEnter: will be called when the interaction flow reaches the interaction step. Parameters are:
    • qiContext: the QiContext.
    • interactionHandler: handles how to end the interaction. Can be resumed/paused/reset at will.
  • onExit: will be called when the interaction step is finishing. Use it to cancel your actions, clean your objects, remove your listeners…
class MyInteractionBehavior : InteractionBehavior {

    override fun onEnter(qiContext: QiContext, interactionHandler: InteractionHandler) {

        // Autonomous stop APIs
        interactionHandler.resumeAutonomousStop()
        interactionHandler.pauseAutonomousStop()
        interactionHandler.resetAutonomousStop()
        interactionHandler.isAutonomouslyStopping

        // Requesting stop API
        interactionHandler.requestStop()

    }

    override fun onExit() {

        // Here, clean your objects, remove your listeners, etc...

    }
}

Interaction Handler

This class contains several features:

  • Autonomous stop:
    • When no interaction criteria are met anymore, the interaction will stop by itself.
    • The criteria are:
      • A human has been seen in the last 10 seconds.
      • A touch has been done on the tablet in the last 10 seconds.
    • Can be resumed/paused/reset at will (paused by default). Resume it with resumeAutonomousStop(), pause it with pauseAutonomousStop(), and reset it with resetAutonomousStop(). Resuming will also reset the criteria status.
    • The current status can be obtained via the isAutonomouslyStopping property.
    • It’s strongly recommended to let it run all the time, except when you specifically need to do some long action during which the interaction shouldn’t stop by itself (example: make Pepper dance, or go somewhere).
  • Requesting stop:
    • Additionally, you can also call the requestStop() method to ask for the library to immediately finish the interaction step, without waiting for the autonomous stop to trigger by itself.

Main activity

Single activity limitation: the whole interaction flow is run into the scope of your chosen activity, so once the flow has begun, starting any other activity will make the flow reset to its initial state when you come back to your chosen activity.

SpeechBar: if you want to use the SpeechBar, don’t forget to make your activity extends the RobotActivity.

GUI handling: the interaction flow displays fragments, so you need to provide a fragment container view to the flow.

The main activity is pretty straight-forward, it just needs to build a Flow, then register to it in the onCreate() method, and unregister from it in the onDestroy() method.

class MainActivity : RobotActivity() {

    /////////////////
    // INTERACTION //
    /////////////////

    private val myInteractionBehavior = MyInteractionBehavior() 

    private val myInteraction = InteractionBuilder()
        .withInteractionBehavior(myInteractionBehavior)
        .build()

    //////////
    // FLOW //
    //////////

    private val myFlow = FlowBuilder(R.id.my_fragment_container_view)
        .withInteraction(myInteraction)
        .build()

    //////////////
    // ACTIVITY //
    //////////////

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        setSpeechBarDisplayStrategy(SpeechBarDisplayStrategy.IMMERSIVE)

        setContentView(R.layout.activity_main)

        myFlow.register(this)
    }

    override fun onDestroy() {

        myFlow.unregister(this)

        super.onDestroy()
    }
}

UI-wise, the layout only needs a fragment container view.

<?xml version="1.0" encoding="utf-8"?>
<androidx.fragment.app.FragmentContainerView xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:id="@+id/my_fragment_container_view"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity" />

How to update the GUI while in interaction

If you want to update the GUI while in interaction, you have to handle it by yourself.

Here is an example of doing so by passing your interaction’s fragment to your interaction’s behavior, and using the ChildFragmentManager of your interaction’s fragment:

Main activity:

class MainActivity : RobotActivity() {

    /////////////////
    // INTERACTION //
    /////////////////

    private val myInteractionFragment = MyInteractionFragment()

    private val myInteractionBehavior = MyInteractionBehavior(myInteractionFragment)

    private val myInteraction = InteractionBuilder()
        .withInteractionBehavior(myInteractionBehavior)
        .build()

    //////////
    // FLOW //
    //////////

    private val myFlow = FlowBuilder(R.id.my_fragment_container_view)
        .withInteraction(myInteraction)
        .build()

    //////////////
    // ACTIVITY //
    //////////////

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        setSpeechBarDisplayStrategy(SpeechBarDisplayStrategy.IMMERSIVE)

        setContentView(R.layout.activity_main)

        myFlow.register(this)
    }

    override fun onDestroy() {

        myFlow.unregister(this)

        super.onDestroy()
    }
}

Interaction behavior:

class MyInteractionBehavior(private val myInteractionFragment: MyInteractionFragment) : InteractionBehavior {

    private val mySecondInteractionFragment = MySecondInteractionFragment()

    override fun onEnter(qiContext: QiContext, interactionHandler: InteractionHandler) {

        myInteractionFragment.displayChildFragment(mySecondInteractionFragment)

    }

    override fun onExit() {

    }
}

Interaction fragment:

class MyInteractionFragment : Fragment() {

    override fun onCreateView(inflater: LayoutInflater, container: ViewGroup?, savedInstanceState: Bundle?): View? {
        return inflater.inflate(R.layout.fragment_interaction, container, false)
    }

    fun displayChildFragment(childFragment: Fragment) {

        activity?.runOnUiThread {

            // Display child fragment
            childFragmentManager
                .beginTransaction()
                .replace(R.id.my_child_fragment_container_view, childFragment)
                .commit()

            // Show child fragment layout
            if (my_child_fragment_container_view.visibility != View.VISIBLE) {
                my_child_fragment_container_view.visibility = View.VISIBLE
            }
        }
    }
}

Interaction fragment’s layout:

<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MyInteractionFragment">

    <androidx.fragment.app.FragmentContainerView
        android:id="@+id/my_child_fragment_container_view"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:visibility="gone" />

</androidx.constraintlayout.widget.ConstraintLayout>

Customization

As described previously, you can also customize the rest of your interaction flow:

class MainActivity : RobotActivity() {

    ////////////////////
    // INITIALIZATION //
    ////////////////////

    private val myInitializationFragment = MyInitializationFragment()

    private val myInitializationBehavior = MyInitializationBehavior()

    private val myInitialization = InitializationBuilder()
        .withFragment(myInitializationFragment)
        .withInitializationBehavior(myInitializationBehavior)
        .build()

    //////////
    // IDLE //
    //////////

    private val myIdleFragment = MyIdleFragment()

    private val myIdleBehavior = MyIdleBehavior()

    private val myIdle = IdleBuilder()
        .withFragment(myIdleFragment)
        .withIdleBehavior(myIdleBehavior)
        .build()

    ///////////
    // START //
    ///////////

    private val myStartFragment = MyStartFragment()

    private val myStartBehavior = MyStartBehavior()

    private val myStart = StartBuilder()
        .withFragment(myStartFragment)
        .withStartBehavior(myStartBehavior)
        .build()

    /////////////////
    // INTERACTION //
    /////////////////

    private val myInteractionFragment = MyInteractionFragment()

    private val myInteractionBehavior = MyInteractionBehavior()

    private val myInteraction = InteractionBuilder()
        .withFragment(myInteractionFragment)
        .withInteractionBehavior(myInteractionBehavior)
        .build()

    /////////////////
    // UNAVAILABLE //
    /////////////////

    private val myUnavailableFragment = MyUnavailableFragment()

    private val myUnavailableBehavior = MyUnavailableBehavior()

    private val myUnavailable = UnavailableBuilder()
        .withFragment(myUnavailableFragment)
        .withUnavailableBehavior(myUnavailableBehavior)
        .build()

    //////////
    // FLOW //
    //////////

    private val myFlow = FlowBuilder(R.id.my_fragment_container_view)
        .withInitialization(myInitialization)
        .withIdle(myIdle)
        .withStart(myStart)
        .withInteraction(myInteraction)
        .withUnavailable(myUnavailable)
        .build()

    //////////////
    // ACTIVITY //
    //////////////

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        setSpeechBarDisplayStrategy(SpeechBarDisplayStrategy.IMMERSIVE)

        setContentView(R.layout.activity_main)

        myFlow.register(this)
    }

    override fun onDestroy() {

        myFlow.unregister(this)

        super.onDestroy()
    }
}

The Interaction Flow Library also provides a Kotlin DSL (Domain Specific Language), that you can use like this:

class MainActivity : RobotActivity() {

    //////////
    // FLOW //
    //////////

    private val myInteractionFragment = MyInteractionFragment()

    private val myFlow = flow(R.id.my_fragment_container_view) {

        unavailable {
            fragment(MyUnavailableFragment())
            behavior(MyUnavailableBehavior())
        }

        initialization {
            fragment(MyInitializationFragment())
            behavior(MyInitializationBehavior())
        }

        idle {
            fragment(MyIdleFragment())
            behavior(MyIdleBehavior())
        }

        start {
            fragment(MyStartFragment())
            behavior(MyStartBehavior())
        }

        interaction {
            fragment(myInteractionFragment)
            behavior(MyInteractionBehavior(myInteractionFragment))
        }
    }

    //////////////
    // ACTIVITY //
    //////////////

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        setSpeechBarDisplayStrategy(SpeechBarDisplayStrategy.IMMERSIVE)

        setContentView(R.layout.activity_main)

        myFlow.register(this)
    }

    override fun onDestroy() {

        myFlow.unregister(this)

        super.onDestroy()
    }
}

License

See the COPYING file for the license.